As far as the speaker's concerned, there's no ambiguity here, and therefore no need to go through what he regards as a pointless ritual of mending the stone wall year after year. Boundary walls are only necessary where it's difficult or impossible to determine which piece of land belongs to whom. But that's not the case here at al—as the speaker points out, his land is all apple orchard, whereas his neighbor's is pine. In a supremely ironic tone, he tells his neighbor that there's no danger of his apple trees crossing over to his land and eating the cones under his pines.

But the neighbor's having none of it. He just repeats the tired old mantra about good fences making good neighbors and carries on mending the wall, heedless of the speaker's concerns. What's more, there's no sense in which the speaker's neighbor is even thinking for himself. As we find out toward the end of the poem, he's just mindlessly regurgitating the words of his father. It's this unthinking devotion to tradition, more than anything else, that annoys the speaker so much.