"We remembered all the young men her father had driven away"
appears in William Faulkner's "A Rose for Emily" in section II, when the unnamed third person narrator tells the story about the time when Emily Grierson's father dies and Emily goes under a fierce stage of denial about her father being dead.
During this time, the narrator explains how Emily's behavior is so odd that she even has the ability to change her entire demeanor, since she is under the complete understanding that nothing has really happened.
The day after his death [...] She told them that her father was not dead. She did that for three days, with the ministers calling on her, and the doctors, trying to persuade her to let them dispose of the body. Just as they were about to resort to law and force, she broke down, and they buried her father quickly.
This is when the narrator explains how, when this happened, they felt sorry for her and even felt the need to understand her situation. Here is a woman who has always been over-protected by her father, kept by him, and even controlled by him only to see him go away, leaving her at her own mercy.
When the townsmen see her breaking down, they realize that perhaps Emily had stayed too long under her father's wing. How could she not cry? She was never able to make real connections with the outside world. Her father's own Southern pride and penchant for over-protection resulted in a plethora of missed opportunities for Emily to get married and have a normal life as a regular lady. She lost her chance to be taken care of by someone during her father's life, and now she is on her own after her father's death. The reason why she held on to her father is because he is all that she had left.
This is why the townsfolk narrator says:
We did not say she was crazy then. We believed she had to do that. We remembered all the young men her father had driven away, and we knew that with nothing left, she would have to cling to that which had robbed her, as people will.