On page 33, Hawthorne describes how Hester lived isolated on the outskirts of town because of her scarlet letter and shame among the townspeople.
"On the outskirts of town, within the verge of the peninsula, but not in cloes vicinity to any other habitation, there was a small thatched cottage. It had been built by an earlier settler, and abandoned because the soil about it was too sterile for cultivation, while its comparative remoteness put it out of the sphere of that social activity..."
On page 148 you could see how Dimmesdale's secret sin has made him feel isolated from his fellow brotherhood as he speaks with Hester in the forest of this.
And as for Chillingworth, he completely gave up his life into the isolation of torturing Dimmesdale's soul everyway he could. We are shown this in chapter 24 when Hawthorne tells us what came of Chillingworth shortly after Dimmesdales death.
"This unhappy man had made......"(Pg.215) continue the whole quote to show its meaning.
Hope this helps!