The crime that Hester commits is that of adultery. The novel does not say how the villagers found out, but given that Hester was pregnant from this "crime" it is easy to assume what gave it a way. This is the reason why her punishment is to carry a letter "A" on her bosom for the rest of her life. As such, her punishment is one of humiliation due to the nature of the crime.
The reader finds out all of this on chapter III, "The Recognition", which is when Roger Prynne, Hester's husband, returns back to society after 2 years of being presumed dead.
After surviving a shipwreck, Roger is rescued by Native Americans and is held captive for a while. Now that he re-enters the Massachusetts Bay Colony, he encounters his (now former) wife standing at the scaffold receiving the scorn from the Puritans. It is here that both, he and Hester, recognize one another. He learns from a townsman that Hester had committed this crime, but that her circumstances render her unable to be punished to the fullest. This is because she is young, new to the village, impressionable but, most importantly, because her husband had been presumed dead in the first place.
“Now, good Sir, our Massachusetts magistracy, bethinking themselves that this woman is youthful and fair, and doubtless was strongly tempted to her fall;—and that, moreover, as is most likely, her husband may be at the bottom of the sea;—they have not been bold to put in force the extremity of our righteous law against her. The penalty thereof is death."
These factors do not necessarily work in favor of Hester. She has to be made into an example, and this is why the humiliation stays. However, keep in mind that she also refuses to give up the name of the father of her child, which makes her doubly guilty of adultery as she is not showing remorse nor doing what she is asked to do. Moreover, what if the husband of any of the "beef and ale", plump goodwives of the village is involved? It is too much at stake to just let her be.
Another problem with Hester is that she was new to the village and was placed under the "spiritual supervision", as it was customary in societies such as the Puritans', of a mentor and leader. This leader was Arthur Dimmesdale; a young, educated, handsome Englishman who was considered a very eligible bachelor.
The novel does not say anything about the relationship between Hester and Dimmesdale until well into the novel when Hester's ex had already infiltrated the village under the name of Roger Chillingworth, and pretends to be a physician who can take care of the now-ailing Reverend Dimmesdale. With the promise he makes Hester make to him of never to reveal his true identity, he is able to penetrate the depths of the minister's mind. It is when he finds out the secret that Dimmesdale has carved a letter "A" into his chest that he discovers the connection between he, Hester, Pearl, and the debilitating condition of guilt that is wearing him down.
Therefore, the reader finds this information first in chapter III with the townsman's revelation, and then through hits throughout the novel where we find that Dimmesdale condition becomes worse in the presence of Hester.