Since the poem is so driven by a mood or a feeling and not so much by a specific purpose or storyline, writing this is going to be both very creatively based and subjective. This is going to mean that you are going to have to summon some imaginative storylines that could simulate why the traveller is there, what was promised in the past, and what conditions brings the traveller to the castle.
Another approach would be to take the poem at its title and compose a piece based on the phantoms that await the traveller. It would be really interesting to hear what promise compelled the traveller to return and why the calls go unanswered. In a world of constant change and mutability, being able to deliver this from the point of view of the phantoms or spirits where the traveller arrives could be interesting.
I think that the construction of this might involve some type of promise made when the traveller was young, and the fulfilment of this promise was his return. Perhaps, there was some type of estrangement between the traveller as a child and the castle in which his family lived. A death in the family, a promise that was exacted out of an elder family member towards whom the traveller held some fledgling notion of loyalty could be where his need to return was evident. I think the idea of something incomplete, some aspect of psychological totality missing in the traveller's makeup is critically important to establish in your narrative. The story is one way to develop this, but I like the idea of the diary entry and a series of diary entries from either or both of the traveller and the home to which he returns could be really interesting. The poem gives you so much from which to work that I think that this can be really something great to develop.