What suggests that the speaker's attitude toward the wall is not necessarily Frost's in "Mending Wall?"
Reading this poem over several times, I looked for something that seems to contradict the general mood of the poem.
The poem is about mending a wall between two properties, something the speaker and his neighbor do each spring. The speaker compares the activity to a game. Philosophically, he also suggests that it seems to be ludicrous to even have a wall because neither man owns animals that might wander without one, and the two different species of trees they grow will not cross the wall or interfere with each other in any way.
Up until this point, the poem has been entertaining. The shift I notice is at the end of the poem. It seems that the wall is there because it has always been there, because the neighbor's father wanted it there. There seems to be no other reason. Frost's writing seems to cast doubt on the wisdom of such an attitude with the following lines:
He only says, 'Good fences make good neighbors'...
He moves in darkness as it seems to me~
Not of woods only and the shade of trees.
He will not go behind his father's saying,
And he likes having thought of it so well
He says again, "Good fences make good neighbors."
The tone with the line "moves in darkness" changes the mood of the poem for me. The sense I get from this line is that the neighbor acts not of his own free will, but because he had heard his father say this many times. In that these men seem to have been rebuilding this wall for many years, perhaps Frost is puzzled that the man cannot, after all this time, decide whether he feels the wall is necessary at all.
On the other hand, perhaps the title, "Mending Wall," is a play on words for "mending fences." Perhaps in the past there was a need for a wall separating the properties and the author simply wonders why the neighbor cannot decide about the wall on his own.
The end of the poem, however, suggests, with the introduction of an image of the man walking in darkness, that Frost may not have the same sense that fixing the wall is like a game, or a time of camaraderie as the speaker seems to infer. It simply may mean that Frost cannot make sense of how the neighbor's mind works.