Mostly his mother--he remembers her and how he treated her with regard to the food and his little sister.  After the disappearance of Winston's father, there was very little food and he demanded more than his share since he was the boy and was always hungry.  He remembers her and his sister going down into the ground...

That relates to his affection for the Proles--they have no fear, they have freedom to do what they want.  They sing--he remembers a song his mother used to sing before his father disappeared.  He is attached to that music.

His present revolves around his lack of knowledge about what happened to them and why he has no family now.  Even a family like those who live in his building would be preferable to none.  His uneasiness and unhappiness stems from these memories.