Chapter VI of The Scarlet Letter "Pearl" is where we first find an indication that there is a rumor going around Pearl's paternity. Rather than mere gossip, the speculation of Pearl's paternity is based on the Puritan idea that any child that has not been officially recognized, or is illegitimate, is no longer a normal child; that the origin of a child who has no traceable origin must be supernatural, specifically, demonic.
Toward the end of the chapter, we learn that the gossip has already reached Hester's ears, and that although she is concerned she is hardly surprised; even Martin Luther had been accused in his time of a similar origin. Hester was not at all shocked at the manner in which the sanctimonious villagers cruelly speak of an innocent child.
She remembered—betwixt a smile and a shudder—the talk of the neigh-bouring townspeople; who, seeking vainly elsewhere for the child's paternity, and observing some of her odd attributes, had given out that poor little Pearl was a demon offspring;
Moreover, the villagers blame Hester, as the mother of such child, for promoting "some foul and wicked purpose" in the commission of her indiscreet "sin".
What these views suggest is that the villagers are extremely set in their ways, in their views, and in the idea that they are doing the right thing by using Hester as their spiritual scapegoat. While they talk about Pearl and Hester, what about Mistress Hibbins's sins? What about Gov. Bellingham hypocritical double standards? What about the so-called sense of mercy and community towards the fellow men? The villagers are obviously low in morals but quite high in judgment. Hester, "betwixt a smile and a shudder" can clearly see what is really going on. She may have committed a sin, but she is way beyond the villagers spiritually and intellectually.