From the poem, we know that the traveler returns to the house to keep a promise to meet.
For he suddenly smote on the door, even/ Louder, and lifted his head:—/‘Tell them I came, and no one answered,/ That I kept my word,’ he said.
Walter De la Mare leaves the rest to our imagination. Who was the traveler looking for and who was he keeping his promise to? We don't know the answers to these questions; that's the beauty of this poem. Each reader who enjoys this poem enjoys it through the prism of his/her own experiences, values, and beliefs.
We can only speculate about the individual/s the traveler hopes to see:
1) Perhaps they are dead, and now haunt the house as living ghosts. The traveler comes too late and does not know he has missed his chance to meet with the pertinent individuals.
2) Perhaps the individual/s in question are inside the house, but for whatever reason, do not wish to answer the door. They are so quiet that the house might as well be inhabited by phantoms.
3) The house is locked, and the owners or occupants have moved away or are temporarily unavailable.
I am sure you can come up with a whole host of other alternatives. However, the very basic reason the traveler returns to the house is to keep a promise to meet with someone who lives there or used to live there. The rest is up to your imagination.
The traveller had promised to come back to meet the people in the house,but he was very late.