One of the reasons why this is an important element for the atmosphere of nostalgia, melancholy, loss, and drastic change of the story, is because Victorian England was a place of extremes: You were either well to do, or broke and miserable. The so-called Middle Class was what we would call these days the Upper Class, while their actual Upper Classes were the aristocrats. So either you belonged to one of those, or you would be in the slum district of the East End.
In David's case, he was moved from the idylic and near surreal world of class, money, and privilege, and was dunked down to the bottom, and now had to understand life from the perspective for both the rich and the poor. When the poor are described in the story, you experience the extreme poverty of the lower classes of that Era. Makes you wonder: Why did this boy have to suffer such extremes? Definately puts you in touch with the struggles of the protagonist, and helps you contrast the different stages of his life.