The nature of their sin is different. Dimmesdale violates his conscience. It is stated emphatically in Chapter 9: " Mr. Dimmesdale was a true priest, a true religionist, with the reverential sentiment largely developed, and an order of mind that impelled itself powerfully along the track of a creed, and wore its passage continually deeper with the lapse of time. In no state of society would he have been what is called a man of liberal views; it would always be essential to his peace to feel the pressure of a faith about him, supporting, while it confined him within its iron framework." When he violated an essential rule of his faith, he was alienated from it and himself.
On the other hand, Hester violated a rule of their society and she willingly accepted the punishment for it --- the Letter. However, it is not at all clear that she violated her conscience. As late as Chapter 17, when she meets Dimmesdale in the forest, she argues that "What we did had a consecration of its own" --- and consecration is a very religious word that suggest more than just neutrality or acceptability of the action.
Chillingworth is destroyed by a sin that is more evil than any sin of passion. Even though he is obsessed with his own quilt, Dimmesdale tells Hester, in Chapter 17, "That old man's revenge has been blacker than my sin. He has violated, in cold blood, the sanctity of a human heart. Thou and I, Hester, never did so!" He understand, as Hawthorne suggests, that guilt is not just one thing, but is a product of upbringing, surrounding circumstances, and differing motives for the same actions.
Hester's sin is worn outwardly, in time forgiven and forgotten by the townspeople (if not by Pearl and herself) whereas the "A" comes to mean "able" instead of "adulterer".
Pearl's sin is inherited through her mother and is therefore also considered an outcast in the community. She is considered to be "otherworldly" which is indicated by her elflike nature and descriptions making her more of the earth, fairy, and supernatural world than of the human world.
Arthur, unable to be open with his sin, suffers most. His is an inward "rotting" so to speak. He wears his sin as a form of physical, spirtual, and emotional sickness which he fears others will recognize.
Hester is changed by her sin, she matures, becomes maternal. Taking care of both her child and others less fortunate in society. She brings food and clothing to the poor. Other women begin to look up to her for her courage and individuality, and for not letting her punishment destroy her. She is an early feminist, fighting a serious case of sexism in her Puritan village. Hester gets stronger as a result of her sin.
Arthur, having suppressed his guilt, over both his sin and the fact that Hester refuses to name her lover, becomes both spiritually and physically sick. He submerges himself in his work, wirting sermons that draw people to him. While the congregation admires him for his spiritual insight, internally, he is spiritually stricken, heartsick. Arthur grows weak from his sin.
Roger develops an evil nature, bent on revenge. He and Hester did not share a loving marriage, and when he returns, all he wants to do is hurt people.
Pearl is a very outspoken child. She is curious about her mother's scarlett letter, asking about it. Hester dresses Pearl in bright colors and does not discourage her daughter's curiosity. Pearl is forceful and direct with Hester, she can be contrary at times, as a result of being left out by the other children.
Hester's sin causes her to be singled out and ostracized--physically and emotionally--from the rest of her community. However, Hester seems to find peace in this outcasting, which was certainly not the norm or the intent.
Pearl inherits her sin from her mother (just as the Puritans believed all people inherited sin and evil from the Fall from Grace in the Garden of Eden). Pearl is judged before people even get to know her based on who her mother is.
Dimmesdale's sin makes him physically ill and creates one of the most conflicted characters in the book. While Hester can openly admit to her sin and 'suffer' through her punishment, Dimmesdale has to uphold his position as preacher. He knows if the community found out that he had committed such a grievous sing, there would be chaos.
Hester's sin turns Chillingworth into somewhat of an obsessed stalker--for her and Dimmesdale. Chillingworth is determined to find out what is going on with her.