The speaker clearly refers to forces of nature in the second line of the poem as being responsible for the wall's deterioration. He mentions that the ground becomes frozen and this causes cracks in the soil on which the wall rests. The constant cracking leads to stones eventually falling off the wall.
"That sends the frozen-ground-swell under it,
And spills the upper boulders in the sun,
And makes gaps even two can pass abreast."
The gaps created are large enough for two persons to pass through, shoulder to shoulder.
Secondly, the speaker blames hunters:
"The work of hunters is another thing:
I have come after them and made repair
Where they have left not one stone on a stone,
But they would have the rabbit out of hiding,
To please the yelping dogs."
The hunters break down the wall to get rabbits out of hiding so that their "yelping dogs" can chase after them. The speaker has constantly come after the hunters and repaired the wall. He realizes that no one has actually seen or heard the wall being broken, so he reasons that these two factors are the most likely reasons for the damage.
It is clear from the speaker's tone that he does not like the wall and has been making futile attempts to convince his neighbor of its uselessness, but his neighbor is adamant that "good fences make good neighbors" and prefers having the wall up. His argument about 'elves' who do not want the wall emphasizes how preposterous the idea of having a wall is, but his neighbor is unmoved.
It is strange that the speaker does not directly confront his neighbor about having the wall removed, perhaps because he does not want to offend him, or maybe because he likes the idea of making contact with his neighbor when they go about fixing the damage.