The wall in the poem can certainly be seen as a metaphor for barriers that exist between people, preventing communication and understanding. One "wall" suggested in the poem is generational. The speaker in the poem seems younger than his neighbor, perhaps because he is more open-minded and less bound by tradition. His neighbor does not understand why doing away with the wall would be a good idea, and the speaker does not understand why his neighbor continues to live by his father's old philosophy. They view the stone wall from different generational perspectives.
Another kind of wall suggested by the poem is that of personality. The speaker possesses a humorous nature. He does not take this yearly ritual very seriously; he sees it as "just another kind of outdoor game." Placing the stones back into the wall is a balancing act, requiring a magic spell, of sorts. The speaker whimsically says to the pesky stones, "Stay where you are until our backs are turned!" He thinks of suggesting to his neighbor that "elves" are responsible for the damage each spring, but he doesn't. He would rather his neighbor thought of it himself, which he never will. His neighbor, in contrast, shows no sense of humor. He goes about his work grimly and methodically, moving like "an old-stone savage."
Many other kinds of walls serve to separate people: for instance, prejudice, poverty, fear, anger, pride, and cultural differences.