Chapter 4 of Nathaniel Hawthorne'sThe Scarlet Letter,titled "The Interview" shows the first first to face communication between Hester and her then-thought to be dead husband Roger Chillingworth.
As Hester suffered the humiliation of standing at the scaffold, Roger Chillingworth anonymously entered the village to infiltrate within and extract, in one way or another, the information regarding who is the man with whom Hester was unfaithful to him.
In Chillingworth's opinion, the double punishment is well-deserved because whoever wronged Hester had wronged Chillingworth as well.
Therefore, as a man who has not thought and philosophised in vain, I seek no vengeance, plot no evil against thee. Between thee and me, the scale hangs fairly balanced. But, Hester, the man lives who has wronged us both!
Although these words are used as an attempt to get Hester to speak up, they do have significance in Chillingworth's heart. Not only is Chillingworth's pride hurt, but his entire life has had to change as a result of Hester's shame. Moreover, the birth of Pearl further reminds him of his own inability to provide Hester with a normal, healthy marriage. He goes as far as to admit that it is he who had wronged Hester
Mine was the first wrong, when I betrayed thy budding youth into a false and unnatural relation with my decay.
Hester is not hypocritical. She is clear about her lack of love for Chillingworth since the very day that they married. As a woman of her time, she had little to no say in her destiny. As a result, it is no surprise that her natural, passionate self took over once she saw herself as a potential widow. All of these are issues that undeniable enrage Chillingworth as both a man and as a husband. Hence, he finds that Dimmesdale needs to be punished for hurting both ends of his pride.