When the speaker and his neighbor begin the yearly ritual of repairing the wall, it seems to be just that - an annual replacing of the rocks that have fallen away or been removed in the past year. On the agreed-upon date, they work together from their respective sides of the wall. "And on a day we meet to walk the line And set the wall between us once again."
It's definitely work, but it's not terribly stressful; it's almost enjoyable to make it all fit together as they move along the way; neighborly cooperation at its best. "To each the boulders that have fallen to each...We have to use a spell to make them balance...another kind of outdoor game."
As they move along, however, the speaker raises the question, why do we need to keep doing this? The speaker points out that they have adjoining fields that will not affect the other's crop, which means the wall really isn't necessary and, in fact, could be perceived as being an insult to their neighborly friendship.
Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offence. Something there is that doesn't love a wall, That wants it down!
The neighbor wants the wall, however. The speaker ends the poem perceiving not a friend and neighbor sharing an annual rebuilding task, but an "old-stone savage" moving in "darkness...not of woods only and the shade of trees." The speaker may not feel threatened by his neighbor, but he certainly doesn't still see him as a friend.