The townspeople were very critical of her choice to see Teacake. From an external point of view, the marriage of Janie and Joe Starks has everything required for success: wealth, respect, a business and stability. For African Americans of this time period, these were nearly impossible goals, and Janie and Joe are proof that blacks can succeed. The people in town never realize that Starks is slowly killing Janie by trying to force her into a role which she never wanted.
So, when she starts to see Teacake, the town sees this as a betrayal. Before, the town has held up Janie as evidence that a black woman could achieve success, but instead of appreciating the wealth and respectablity that Joe provided, she is giving that up for "love." Many of them believe that Janie is too sexual, giving up the responsibilities of an adult woman for the frivolous and silly infatuation of a girl. This is why, when Janie returns, the women want to hear all the gossip. They assume that Janie has come home because Teacake has betrayed her. They see this as suitable punishment for her unwillingness to follow the social rules.
That conflict is central to the book. The analysis link below explains the metaphor of the ship offered in the very first chapter. The narrator explains that men wait for their ship to come in, meaning they are at the mercy of luck. Women, however, “forget all of those things they don’t want to remember, and remember everything they don’t want to forget.” According to the link, that means that they control their own lives. Janie has taken control of her own life, but the people in town will always see her as someone who turned her back on them and their community.