Although the reason for the departure of the Drover family home is not explicitly referenced, we can infer the reason, especially if we consider the description of the house that we are given and the reference that is made to bomb damage. Consider the following quote:
There were some cracks in the structure, left by the last bombing, on which she was anxious to keep an eye. Not that one could do anything--
Such information leads us to infer that this house has been abandoned because it is in London and the story is set during the Second World War, when London, as well as many other British cities, were blitzed by the German air force. Many people left London as a result, and the Drover family was just one of these many families who moved to a country home for safety for the duration.