Hester Prynne does not speak to Roger Chillingworth until chapter four of The Scarlet Letter by Nathaniel Hawthorne--and she undoubtedly would not speak to him then if she did not have to. She is sitting in a prison cell as part of her punishment for being caught as an adulteress, and her baby is upset. Fortunately, there is a doctor who is also staying in the prison, though he is not a prisoner; his name is Roger Chillingworth and he is Hester's husband. He is a doctor, and he comes to Hester's cell to soothe the child.
Chillingworth (who is calling himself by this name, though his name must have been Roger Prynne since that is Hester's name) sent Hester, his young wife, over to New England. He was going to follow soon after but ended up as a prisoner of the Native Americans and was of course delayed in his arrival in Boston. When he does arrive, imagine how stunned he must have been to see his wife standing on the public scaffold with a child, being punished for her adultery.
Imagine, too, how frightened and even ashamed Hester must have been when she faces her husband for the first time. There is no hiding her sin, and surely there is nothing she can say to make things right between them. Fortunately for her, Roger understands and says:
"I seek no vengeance, plot no evil against thee. Between thee and me, the scale hangs fairly balanced."
If the baby were not crying inconsolably this night in the prison cell, the meeting between Hester and Roger might have been delayed; however, it is certain they would have had this conversation sometime and in some place.