The first thing to consider is that, in The Scarlet Letter, isolation comes as a result of sin. Hester's isolation by imprisonment and her subsequent banishment to the far side of the settlement are the result of her sinful act with a then-unknown man. Dimmesdale's isolation is not physical, but Hawthorne cleverly denotes how sin isolates Dimmesdale from the person whom the world had made him out to be; he also becomes isolated from his faith, his integrity, and his self-image as a godly man.
Similarly, it is Hester's sin that prompts the changes and sacrifices made by Chillingworth which, eventually, will change and end his life.
In chapter 3, we know that Chillingworth is actually a learned man who led a fruitful life in England. On his way to Amsterdam he lets his wife come to America first, but his boat capsizes and he is presumed dead. However, as Hester's current condition is known, Chillingworth decides to make the ultimate sacrifice, albeit, not for a saintly purpose: has come incognito to visit Hester and to obtain revenge for her committing adultery. In other words, he completely ceases life, as he knows it, and changes forever. He changes his name, enters the life of the settlement, and inspires the trust of the aldermen in order to continue his vengeful plan.
Unknown to all but Hester Prynne,... he chose to withdraw his name from the roll of mankind, and, as regarded his former ties and interest, to vanish out of life as completely as if he indeed lay at the bottom of the ocean, whither rumor had long ago consigned him.
Chapters IX and X, named "The Leech" and "The Leech and his Patient" respectively, focus on Chillingworth's hunger for revenge and how it continued to isolate him so greatly from the rest of society that even his looks become those of an otherworldly being. He has, after all, dedicated his whole existence to the mere duty of finding out Hester's secret and, later, to make Dimmesdale life quite torturous.
Therefore, it is his physical change that prompts the other men in the settlement to start having second thoughts about Chillingworth's worthiness as a physician, and as a citizen. His aspect, brought in by giving himself like sacrificial lamb for the task of punishing Hester, awoke in others feeling of distrust, and even fear.
A large number...affirmed that Roger Chillingworth’s aspect had undergone a remarkable change while he had dwelt in town... At first, his expression had been calm, meditative, scholar-like. Now, there was something ugly and evil in his face, which they had not previously noticed, and which grew still the more obvious to sight...
Similarly, Dimmesdale undergoes tremendous physical changes to include his paleness, his sunken face, his weak demeanor, and the horrid look of pain in the face. The reader knows that his sacrifice comes from self mutilation. However, it is the need for redeeming a sin what isolates Dimmesdale and Chillingworth and join them in a dysfunctional and evil relationship.
Concisely, Chillingworth's isolation is like a sacrificial rite. He offers what is left of his life toward the commission of revenge, and consumes himself until the end when, finally, Dimmesdale dies and confesses to the sin that he committed with Hester.