The narrator's point of view is shaped by idealism. As he drives past the woman and her daughter everyday, he imposes his own vision of who they are and how they live. Therefore, he sets himself up for disappointment because his impressions of the woman and her daughter are not based in reality, but from an image that he creates to suit his purpose.
The point of view expressed by the train conductor regarding the woman and her daughter loses its significance for the narrator after he confronts them. But by that time, he no longer needs them to live in his idealistic vision, he is retired, he has stepped off the train and will no longer gaze upon them as he passes their small cottage everyday.
This story is about nothing but point of view. Every day the engineer has these idealistic visions and beliefs about people he does not even know. He gives these women lives and personalities that they don't really have and don't really deserve. When he knows he's finished working, he decides to greet them and perhaps thank them.
When he gets to their house, he finds that things aren't always as they seem. His point of view changes completely by distance. He sees this woman and her daughter up close. He finds that they are almost hostile towards him and very distrusting daily waving made him feel comforted and bound by a sort of friendship. However, once he sees the reality of the situation, he realizes how idealistic he made it out to me. The reality shattered his hopes. This story shows us that point of view is very specific. If we are too far away, we only see what we want to see. It's hard to avoid reality when we are up close and engaged in it.