The two men in this poem joke about what it is that has caused the gaps in the wall that separates their two properties. Physically, the wall is made of stones, and therefore it needs to be periodically repaired. The speaker suggests that the gaps are made by elves, but, more likely, they are simply places where stones have become dislodged by tree roots, bad weather, animal behavior, and so on and so forth.
Metaphorically, of course, this poem is also working on another level. It might be tempting to interpret the "gaps" in the wall as inroads, places through which intimacy could creep into the relationship between the two men, but it is clear that this wall, in fact, is unifying for them. The neighbor believes that the existence of the wall, a finely demarcated line between their two properties, is necessary for keeping a good relationship between the two "good neighbors." As such, in meeting every year to walk the wall and repair the gaps, the two men are actually contributing to the upkeep and stability of their neighborly relationship. It is not a close relationship, but it is one which is kept positive because it is defined by rules, and the firmly-ruled line represented by the wall cannot have gaps left in it.