From the beginning of the poem we know that the wall between the speaker and the neighbor has somewhat fallen down and needs to be rebuilt.  This wall has two meanings: one is the physical wall, and the second is the emotional wall that the neighbor has built so that he is not emotionally close to the speaker.  The speaker resents that, and that brings up the concept of "belonging."  There is not sense of belonging if there is either a physical OR emotional wall between the two men.  The neighbor says to the speaker

"Good fences make good neighbors."

He doesn't need or want friendship with the speaker, and that angers the speaker.  He views him differently after hearing those words 'Good fences make good neighbors.' He sees him as a savage.

"like an old-stone savage armed.
He moves in darkness as it seems to me"

There is no friendship if there is a wall.  Neither man can belong with or to the other man if the wall remains (the emotional wall in this case).