Well, the speaker doesn't exactly want the wall down: "Something...wants it down."
The poem shows two neighbours coming together for the annual rebuilding of the wall between their properties. Although the speaker seems dubious about what function the wall has, it is he who initiates the rebuilding ("I let my neighbour know...") as well as doing irregular repairs on his own initiative ("I have come after them [hunters] and made repair"). But he also dismisses the rebuilding as "just another kind of outdoor game." Why does he play?
I think the answer is that while he may feel the wall should be unnecessary, he also needs to keep his neighbour at a distance. He teases the man, but is wary of him and cannot understand his clinging to "his father's saying" about good neighbours needing good fences. Near the end, the narrator slips into a defensive tone,
I see him there
Bringing a stone grasped firmly by the top
In each hand, like an old-stone savage armed.
He moves in darkness as it seems to me,
Not of woods only and the shade of trees
This evokes the original function of walls, from the Garden of Eden onward, to keep "darkness" outside, away from human civilization.
To sum up, nature and growth work against walls, but the ever-present threat from the incomprehensible otherness of our fellow humans preserves walls as a guarantee that we know our bounds and will keep our peace by respecting each others' limits.