Chillingworth says that he will not take any kind of revenge on his wife, Hester, for her marital infidelity because, if he were interested in revenge, allowing her to live with the shame of the scarlet letter would be the best way to get it. He says, "'Even if I imagine a scheme of vengeance, what could I do better for my object than to let thee live, -- than to give thee medicines against all harm and peril of life, -- so that this burning shame may still blaze upon thy bosom?'" Killing Hester would only result in the shortening of her punishment; in having to wear the scarlet letter, she will be punished for a long time and in a quite painful way. If he blamed her, this would be the best way to punish her.
Further, he asks, "'Misshapen from my birth-hour, how could I delude myself with the idea that intellectual gifts might veil physical deformity in a younger girl's fantasy?'" Chillingworth knows that he really shouldn't have married Hester in the first place. He was significantly older, deformed, and quite incapable of satisfying a much younger woman's fancy. He realizes that he should never have expected her to be happy with him; of course she would look elsewhere for her happiness. Thus, he considers the scale between the two to be balanced: he wronged Hester when he married her, and she wronged him when she had an affair with another man.
Finally, Chillingworth is most interested in seeking revenge on the man whom he believed has wronged them both: Hester's co-sinner. He says, "'Hester, the man lives who has wronged us both! Who is he?'" Hester will not reveal this man's name, and so Chillingworth makes it his mission to seek out and torture this man.
While Chillingworth does feel some responsibility in having married a young, naive girl, he is more intent on finding who his wife had an affair with. For him, being able to watch his wife be humiliated in the community, while sparing himself the agony of being known as the wronged husband , is enough for him.
Chillingsworth will not take vengeance upon Hester because he can think of no better punishment than that she wear the scarlet "A" for the rest of her life. Also, he takes a little of the responsibility for her sin upon himself, saying he should have never thought that it could work for an old, misshapen man like himself to have married a young and vibrant girl like Hester.
Chillingsworth will not take vengeance on the innocent child either, satisfying his thirst for revenge by plotting against Hester's partner in sin, Dimmesdale.