Hawthorne gives us the answer to this question in Chapter 17 when Arthur says to Hester:
We are not, Hester, the worst sinners in the world. There is one worse than even the polluted priest! That old man's revenge has been blacker than my sin. He has violated, in cold blood, the sanctity of a human heart. Thou and I, Hester, never did so!"
Coming from Dimmesdale, this is an amazing confession. He is almost destroyed by guilt for his violation of his nature; despite his inability to forgive himself, he is able to construct a hierarchy of guilt where sins of passion, sins of the flesh, are far less serious than sins of the intellect --- in this case revenge. Although he may have been wrong about a lot of things, he was correct about this one.
Unlike Chillingworth, Arthur Dimmesdale was an introspective person who continually examined the state of his own soul. Betraying his spiritual beliefs and moral principles caused him great shame, grief, and guilt. Dimmesdale came to loathe himself, not only for his initial sin but also for his weakness in not acknowledging it before God and his parishoners. Arthur longed for salvation and blamed no one but himself for his suffering. He bore no malice toward any other human being. He felt a special contempt for his sins because he was a minister, always aware of his spiritual responsibilities to his congregation.
We can infer that Chillingworth examined his own sins and attempted to atone for them at the end of his life because he left his lands in England to Pearl before he died. However, throughout the novel he sought to salvage his wounded ego as Hester's humiliated husband by revenging himself upon her partner in sin. His revenge was cold, deliberate, and obsessive. He took joy in Dimmesdale's suffering. And unlike Dimmesdale, Chillingworth felt no responsibilities to his vocation; he violated without hesitation the most basic principles of being a physician. He used his skills not to relieve pain, but to inflict it--not to heal, but to destroy. Dimmesdale was morally weak, but Chillingworth was morally corrupt. He chose to violate "the sanctity of the human heart."
Hawthorne was writing The Scarlet Letter more than 100 years after the Puritans reigned supreme in New England. In many of his writings including The Scarlet Letter and "Young Goodman Brown", Hawthorne's point was to prove the hypocrisy of Puritan ideology. This was a belief where it was never possible to be good enough.
Roger Chillingworth sets himself somewhat as a moral superior to Hester and Dimmesdale. To the modern reader, Chillingworth fails to see any failings that he might have in this matter as relevant. As the character of Dimmesdale is gradually revealed to us, we see that what Hester Prynne has suffered outwardly, Dimmesdale has suffered inwardly.