Chillingworth believes that the shame of public humiliation is ample punishment for Hester. The scarlet letter she wears is enough to satisfy him.

When they meet and he gives her a medicinal draught, she fears he is trying poison and kill her in revenge for her infidelity. He insists this is not so, saying:

Live, therefore, and bear about thy doom with thee, in the eyes of men and women,—in the eyes of him whom thou didst call thy husband,—in the eyes of yonder child!

Public shame is terrible to Chillingworth, which is why he wants to keep his marriage to her a secret. He also acknowledges that he is partly to blame for her state, saying:

It was my folly, and thy weakness. I,—a man of thought,—the bookworm of great libraries,—a man already in decay, having given my best years to feed the hungry dream of knowledge,—what had I to do with youth and beauty like thine own!

He understands that Hester would be attracted to a younger and more handsome man.

There is more than a bit of sexism in Chillingworth's attitude toward Hester: shame is enough for a woman, and a woman is inherently a weak vessel who could be expected to fall in the absence of her husband and the presence of an attractive male. Therefore, Chillingworth sets his revenge sights on a worthier opponent: the man who caused Hester's disgrace.

Chillingworth says that he will not take any kind of revenge on his wife, Hester, for her marital infidelity because, if he were interested in revenge, allowing her to live with the shame of the scarlet letter would be the best way to get it.  He says, "'Even if I imagine a scheme of vengeance, what could I do better for my object than to let thee live, -- than to give thee medicines against all harm and peril of life, -- so that this burning shame may still blaze upon thy bosom?'"  Killing Hester would only result in the shortening of her punishment; in having to wear the scarlet letter, she will be punished for a long time and in a quite painful way.  If he blamed her, this would be the best way to punish her.

Further, he asks, "'Misshapen from my birth-hour, how could I delude myself with the idea that intellectual gifts might veil physical deformity in a younger girl's fantasy?'"  Chillingworth knows that he really shouldn't have married Hester in the first place.  He was significantly older, deformed, and quite incapable of satisfying a much younger woman's fancy.  He realizes that he should never have expected her to be happy with him; of course she would look elsewhere for her happiness.  Thus, he considers the scale between the two to be balanced: he wronged Hester when he married her, and she wronged him when she had an affair with another man. 

Finally, Chillingworth is most interested in seeking revenge on the man whom he believed has wronged them both: Hester's co-sinner.  He says, "'Hester, the man lives who has wronged us both!  Who is he?'"  Hester will not reveal this man's name, and so Chillingworth makes it his mission to seek out and torture this man.

While Chillingworth does feel some responsibility in having married a young, naive girl, he is more intent on finding who his wife had an affair with. For him, being able to watch his wife be humiliated in the community, while sparing himself the agony of being known as the wronged husband , is enough for him.

Chillingsworth will not take vengeance upon Hester because he can think of no better punishment than that she wear the scarlet "A" for the rest of her life.  Also, he takes a little of the responsibility for her sin upon himself, saying he should have never thought that it could work for an old, misshapen man like himself to have married a young and vibrant girl like Hester.

Chillingsworth will not take vengeance on the innocent child either, satisfying his thirst for revenge by plotting against Hester's partner in sin, Dimmesdale.