According to the poet, it seems that hunters come by the wall during the winter and remove stones in order to flush rabbits out of their hiding places within the wall. By tearing down sections of the wall, the hunters destroy the rabbits' hiding places, thus allowing their dogs to chase them more easily.
The subtle use of language in "I have come after them and made repair" (line 6) indicates that this is a regular occurrence. The hunters have destroyed sections of the stone wall in the selfish pursuit of their prey and apparently without concern for the landowners, who must work to repair the wall each spring.
It is clear that the narrator does not think highly of the hunters who deliberately destroy the wall. These hunters are unlike the natural forces of destruction, the freezing and thawing of the ground, which destroys parts of the wall as well. The hunters, though, work in concert with these natural forces of destruction. Like the mysterious "something" in the poem's first line, the hunters also don't "love a wall."