The speaker is deep into the woods when he speaks these lines. He says that he knows the person to whom this land and trees belong, but that person's home is far away, in "the village." Consequently, he will not see the narrator and his horse stop to enjoy the quiet and watch the snow fall. In fact, the speaker is deep enough into the woods that there is no one else living anywhere nearby—he says that no "farmhouse [is] near"—and this adds to the peacefulness and tranquility he feels watching the "downy flake."
In the first line of the last stanza, he describes the forest as "lovely, dark and deep," again reinforcing how vast the woods here are and how far into them he has traveled. He has "miles to go" before he can stop, perhaps because there is, quite literally, nowhere for him to sleep without a farmhouse nearby; this fact confirms the depth of the woods as well.