Chillingworth says that he will not take any kind of revenge on his wife, Hester, for her marital infidelity because, if he were interested in revenge, allowing her to live with the shame of the scarlet letter would be the best way to get it. He says, "'Even if I imagine a scheme of vengeance, what could I do better for my object than to let thee live, -- than to give thee medicines against all harm and peril of life, -- so that this burning shame may still blaze upon thy bosom?'" Killing Hester would only result in the shortening of her punishment; in having to wear the scarlet letter, she will be punished for a long time and in a quite painful way. If he blamed her, this would be the best way to punish her.
Further, he asks, "'Misshapen from my birth-hour, how could I delude myself with the idea that intellectual gifts might veil physical deformity in a younger girl's fantasy?'" Chillingworth knows that he really shouldn't have married Hester in the first place. He was significantly older, deformed, and quite incapable of satisfying a much younger woman's fancy. He realizes that he should never have expected her to be happy with him; of course she would look elsewhere for her happiness. Thus, he considers the scale between the two to be balanced: he wronged Hester when he married her, and she wronged him when she had an affair with another man.
Finally, Chillingworth is most interested in seeking revenge on the man whom he believed has wronged them both: Hester's co-sinner. He says, "'Hester, the man lives who has wronged us both! Who is he?'" Hester will not reveal this man's name, and so Chillingworth makes it his mission to seek out and torture this man.