I'd like to start by saying that readers can't know that the house is old and/or deserted.  There are lines that suggest that the house is old and empty, but those lines don't guarantee that fact.  

Let's start with old.  I believe that the best indication that the house is old is the mentioning of the turret.  A turret is reminiscent of old castles, but the design and use of turrets gained popularity again during the Victorian era.  That period of history covers the second half of the 1800's.  The problem is that readers don't know a specific date for the poem's setting.  It's possible that the home was built in 1860, and the traveler in the story is visiting the home in 1880.  I wouldn't call a 20 year old home old.  

The poem does tell readers that birds flew out of the turret, and that the windows are "leaf-fringed." Both of those details make it seem like nature is taking back the home.  That does take some time, but even homes that have sat empty for even as little as a year are subject to ecological succession taking back the house.  I do believe that the house is old because it makes it creepier and more haunted feeling; however, I could defend the idea that the house is newer and has been recently deserted.  

As for the home being empty, I believe that is easier to support.  The narrator tells readers that the halls are empty.  Silence is a repeated motif in the poem.  Reader attention is drawn to the fact that the house is unusually quiet and still.  That suggests to me that the house is devoid of inhabitants.  The use of the word "phantom" also alerts readers to the lack of living souls in the house.  

But only a host of phantom listeners   

   That dwelt in the lone house then 

The words chosen by Walter de la Mare tell us that the house of "The Listeners" is old and deserted, uninhabited by any living person. Let's look at some of his lines. 

First, we can infer from a few descriptions of the house that it is old. We learn that "a bird flew up out of the turret" (line 5) and that as the Traveller mounted his horse as he left, there was "the sound of iron on stone" (line 34). We can infer that because the house has a turret, it is likely to be old. The stone paving leading up to the house suggests an old house as well. Additionally, there is "the leaf-fringed sill" (line 10), which would hardly be leaf-fringed if it were a newer house.

There is a great deal of evidence to show that the house is deserted by living humans. When people occupy a house, they see to it that there are no birds in the house that could fly out of a turret, and they clear climbing vines off the windowsills. No one answers, even when the Traveller persists in knocking. The house is described as "lone" (line 14), and "empty" (line 18). Anything in the house is ghostly or "phantom" (line 13). The house is abandoned by all living people, and the forest seems to be reclaiming it.