What is key to realise is that these phrases occur as part of the story where the narrator strives to understand the mystery of the mansion that so attracts him. The strange woman who he meets who looks after the children who live there deliberately uses the phrases, with ironic meaning, to describe the other-worldly existence that both she and the children occupy. However, it is only at the end of the story that both we and the narrator understand the true import of such phrases.
Initially, we take such phrases as the ones you have identified to refer only to the isolated nature of the mansion and the way it is so far from human society. Remember that the narrator took a wrong turn to arrive there and that the drive was very long. The house itself is "besieged" on three sides by the "marshalled woods," which again reinforces this impression of the mansion being almost swallowed up by nature and removed from the society of man. Being "out of the world" and the references to the woods indicate the way in which not only the house is isolated and separate, but that literally the ghost children that live there operate on a different spiritual plane. It is only at the end, when the narrator finally confronts the children and is kissed on the palm of his hand, that he realises the truth and the two planes of existence that comingle in this house and setting. Knowing the conclusion, we are able to go back and read such remarks as you have highlighted with greater understanding.