In Chapter 4 of The Scarlet Letter, when Chillingworth first encounters Hester after his sojourn in the forest, his attitude toward Hester is one of a dual regret mixed with an ironic sarcasm stirred by an unquenchable, bitter curiosity to know the name of the man who transgressed against Hester and Chillingworth himself. After giving Hester an Indian herbal potion to calm her fevered emotions, Chillingworth addresses her explaining what he thinks and what he wants.
Chillingworth says he holds no blame to Hester's account, though with ironic sarcasm he touches the scarlet letter on her bosom, and says he wants no vengeance and plans no evil against her. He further says he judges that the scales are balanced between them.
Chillingworth takes the primary blame upon himself, accusing himself of taking advantage of her youth and beauty by marrying her to warm his own cold heart even though she made it clear that she felt no love toward him. Chillingworth makes it clear his attitude toward Hester hinges on his being the one who wronged her first.
He further expresses his attitude when he asks who the man was who brought Hester to an adulterer's infamy, saying that that man had "wronged us both!" Chillingworth with this makes it clear that he his attitude toward Hester is that she was wronged, that she was a victim and not a villain though his gestures and asides ("burning shame may still blaze upon thy bosom") make his bitterness clear.