The speaker clearly refers to forces of nature in the second line of the poem as being responsible for the wall's deterioration. He mentions that the ground becomes frozen and this causes cracks in the soil on which the wall rests. The constant cracking leads to stones eventually falling off the wall.
"That sends the frozen-ground-swell under it,
And spills the upper boulders in the sun,
And makes gaps even two can pass abreast."
The gaps created are large enough for two persons to pass through, shoulder to shoulder.
Secondly, the speaker blames hunters:
"The work of hunters is another thing:
I have come after them and made repair
Where they have left not one stone on a stone,
But they would have the rabbit out of hiding,
To please the yelping dogs."
The hunters break down the wall to get rabbits out of hiding so that their "yelping dogs" can chase after them. The speaker has constantly come after the hunters and repaired the wall. He realizes that no one has actually seen or heard the wall being broken, so he reasons that these two factors are the most likely reasons for the damage.
It is clear from the speaker's tone that he does not like the wall and has been making futile attempts to convince his neighbor of its uselessness, but his neighbor is adamant that "good fences make good neighbors" and prefers having the wall up. His argument about 'elves' who do not want the wall emphasizes how preposterous the idea of having a wall is, but his neighbor is unmoved.
It is strange that the speaker does not directly confront his neighbor about having the wall removed, perhaps because he does not want to offend him, or maybe because he likes the idea of making contact with his neighbor when they go about fixing the damage.
Natural causes destroy the wall, as well as man-made problems like hunters.
When the speaker says “Something there is that doesn't love a wall,” he is noticing that it is very hard to keep a wall up in nature. There will always be forces of deterioration that will bring it down. He uses this to justify the fact that there should not be walls, even though his neighbor believes that, "Good fences make good neighbors."
Something there is that doesn't love a wall,
That wants it down.' I could say 'Elves' to him,
But it's not elves exactly, and I'd rather
He said it for himself.
The speaker’s point is that the wall is not needed. He does not see a reason to separate himself from his neighbor, since there is no livestock that is going to do damage. The neighbor seems to think that if there is a division between the two, they will get along better.