In chapter 17 of The Scarlet Letter, "The Pastor and His Parishioner" Hester famously tells Arthur Dimmesdale the true identity of the man with whom he has been sharing his dwelling. She also warns Dimmesdale against "that man", who is crushing the soul right from within Dimmesdale's weakened body.
“Thou must dwell no longer with this man,” said Hester, slowly and firmly. “Thy heart must be no longer under his evil eye!”
Aside from the obvious conflict, another reason why Hester is very concerned about Dimmesdale and Chillingworth living together is that the dwelling that they share symbolizes being trapped inside your own coffin; it is like being buried alive. If Chillingworth is administering Dimmesdale with every tool possible for the latter to become demoralized and traumatized, imagine how fast the death of Dimmesdale will come. It is being trapped with the enemy.
As early as chapter 11 we have a good idea of how badly this relationship has deteriorated:
While thus suffering under bodily disease, and gnawed and tortured by some black trouble of the soul, and given over to the machinations of his deadliest enemy, the Reverend Mr. Dimmsdale had achieved a brilliant popularity in his sacred office.
What this shows is that, far from being the sought-after physician that the villagers thought would come and save their beloved divine, Chillingworth is like a deadly tumor that intends to shut the younger man down until his end comes. We know that this is true because, shortly after Dimmesdale's death, Chillingworth was left with nothing else to live for, and thus he also dies not too long afterwards.