We know that the two farmers in the poem are walking along either side of the wall which divides their two properties, putting it to rights. The narrator is unsure of what exactly they are walling out, but his neighbor is wedded to the idea of this wall between them. The wall is "between us as we go," and we know it is a wall of some size because some of the bits that fall out of it are the size of "boulders." Therefore, we can assume that there is at least a significant thickness of wall between the men.

The farmers are not too far apart to be able to talk to each other. However, the way the narrator describes his companion—"I see him there"—implies that he is at some little distance, particularly because the narrator says it is as if he is moving "in darkness."

There is also, of course, a significant gulf of difference in attitude between the two farmers. While the narrator does not see any real need for a wall, as there are no cows or other animals to be walled out, the other farmer seems to believe that there is some deeper reason why "fences make good neighbors."