I agree with the previous educator's answer. I do not think that the poet has an intentional message or moral to this particular poem. Walter de la Mare's eNotes biography page states that he was "one of the great masters of the supernatural story," which is absolutely true of "The Listeners." It brings up the possibility that the current occupants of the house are actually supernatural beings:
But only a host of phantom listeners
That dwelt in the lone house then
Stood listening in the quiet of the moonlight
To that voice from the world of men
In addition to that, the poem raises other questions about the current occupants, the former occupants, the man, and his reason for being there in the first place. The poem is quite eerie in its mood, and it does a successful job of unnerving the reader. If I was forced to come up with some kind of message that this poem might be conveying, I think I would have to state a message that incorporates the supernatural aspects of the poem. It is possible that the poem's message is that the supernatural might exist, and it does not want anything to do with us.
"The Listeners" by Walter De la Mare is really not a poem with a "message." Walter de la Mare, in fact, would have considered works with "messages" simply bad poetry rather than genuine works of the imagination. "The Listeners" is a poem intended to evoke a mood and setting rather than convey some ideological point.
The protagonist of the poem arrives at a deserted house in the dark. The house appears to be in a remote, rural setting. He knocks on the door and calls out, but no one answers, and eventually he leaves. The "listeners" of the title do not appear to be living human beings. Perhaps they are ghosts, or perhaps they merely are memories. The point of the poem is the uncanny nature of the experience.
As readers, we are not meant to think of this as a puzzle to be solved, but rather to empathize with the sense of mystery and melancholy.