There is a paragraph in chapter 13 that starts with the words:
"The effect of the symbol..."
that clearly delves into an answer for your question. Put simply, the mark of shame that she bore, the scarlet letter, robbed her of her beauty. The paragraph goes on to express that this is the case with all women who encounter something similarly so severe. Hawthorne expresses a truth about women: often events happen to cause them to lose their tenderness. This was Hester's moment of loss. Losing this tenderness can render a woman dead, or it can be beaten out of her. To bring back this representation of what it means to be a woman Hathorne wrote:
She who had once been woman, and ceased to be so, might at any moment become a woman again, if there were only the magic touch to effect the transfiguration.
Hester required a special touch of tenderness to be able to return to that beauty in a woman that defines her. This touch might be the touch of a man, a certain man.