This poem is about an unnamed Traveler who, with his horse, arrives at a seemingly deserted house in the middle of a forest. We know that it is night time ("moonlit") when the Traveler knocks on the door of the house, calling out to see if anybody is there.
By all signs, the house is empty. An owl flies up out of it, suggesting abandonment, and nobody answers the Traveler. The Traveler, to be certain, knocks again on the door to no response.
However, the poet tells the reader that the house is not really empty—instead, a "host of phantom listeners" stand in the house, listening to the voice of the Traveler as if it were a voice from another world —"the world of men." We are not told who these listeners are, but they are evidently eerie and otherworldly, presumably supernatural.
The Traveler seems to perceive their presence as well, for he continues, "Tell them I came, and no-one answered." He is particularly concerned that they pass on the message that he has "kept his word." The poem's mystery is a large part of its charm—we do not know what promise the Traveler made, which he is now keeping as he returns to this abandoned house where only ghosts are here to listen. But by coming to the house, he seems to have fulfilled what was promised. At the end of the poem, the Traveler gallops away into the distance, with the Listeners still listening until they can no longer hear the sound of his horse's hooves.