The speaker doesn't like the wall. He begins the poem by stating,
Something there is that doesn't love a wall,
That sends the frozen-ground-swell under it,
And spills the upper boulders in the sun;
And makes gaps even two can pass abreast.
He feels that nature itself seems to disapprove of the wall, because it does things to break the wall down, to weaken it, and to open up gaps in it. The wall naturally sort of crumbles as a result of the weather and the natural passage of time. Rebuilding the wall just doesn't seem worth the trouble, to the speaker, because his
[...] apple trees will never get across
And eat the cones under his [neighbor's] pines [...]
There is nothing in need of keeping in or keeping out, since the two men only have trees on their respective properties. Further, the speaker says that before he would build a wall, he would like to know
[...] to whom I was like to give offense.
He may feel just a bit offended by his neighbor's stubborn insistence on maintaining the wall. His neighbor simply repeats the adage, "'Good fences make good neighbors.'" The implication, here, is that a neighbor isn't a good one unless he is kept from one's property (and, by implication, one's life) by such a wall. The speaker doesn't feel the need for physical boundaries or distance, but the neighbor evidently does, and so it seems as though the speaker may be precisely what the neighbor wishes to keep at a distance. I'd be offended too!