What about Hester's appearance and "crime" seems to bother the community, particularly the women, the most?
Hester's Appearance: Hawthorne describes Hester as a striking woman, someone's whose countenance commands attention. She also has a purity about her exterior that is especially bothersome to the community in light of her "sin." Moreover, when Hester stands before the community to receive her punishment, she has embroidered a beautiful symbol of shame for herself which impliesto the crowd gathered around the scaffold that Hester is rather defiant, and yet they cannot help but admire her handiwork. Finally, women are stereotypically critical when it comes to someone whom they view as more attractive than themselves. Hawthorne is stressing that stereotype in the women's reaction to Hester's appearance.
Hester's "Crime": The townspeople know that because Hester is with child that she had a partner in her sin. Her husband, as far as they know, has still not arrived in Boston, and so Hester's "partner" lurks among them, seemingly unpunished for his sin. For the women, Hester's lover could be their husbands, fathers, brothers, etc.; so their suspicions rum rampant, and they turn their scorn toward Hester who might have stolen away one of their loved one's affections.