But the minister repulses him, and he calls again to Hester to join him on the scaffold.  As they mount the scaffold "Old Roger Chillingworth" follows them:

"hadst thou sought the whole earth over, " said he looking darkly at the clergyman, "there was no one place so secret,--no high place nor lowly place, where thou couldst have escaped me,--save on this very scaffold!"

The three are united and whole on the scaffold, but Roger Chillingworth is alone.  He kneels down beside the Reverend with a blank,

"dull countenace, out of which the life seemed to have departed."

"Thou hast escaped me!" he repeated more than once.  "Thou hast escaped me!"

Another chapter in which Chillingworth is presented as a dark, alienated character is Chapter 14 in which Hester talks to Chillingworth on behalf of Mr. Dimmesdale.  As she looks steadily at the old man, she is struck with wonder to observe what a change had taken place in him.  As he speaks to Hester, he admits that he has been

"A mortal man, with once a human heart, [who] has become a fiend at his elbow!....

The unfortune physician, ...lifted his hands with a look of horror, as if he had beheld some frightful shape, which he could not recognise, ursurping the place of his own image in a glass.

Then, in Chapter 15, after Hester walks away, Chillingworth, now alone, gathers herbs.  As he does so, Hester wonders at what appears to be

...a circle of ominous shadow moving along with his deformity, whichever himself way he turned himself?

Perhaps the most spiritually isolated character of Hawthorne's novel, Roger Chillingworth has been set apart from his fellow man from his youth when he was a student.  Vowing revenge upon Dimmesdale and determined to learn his secret after returning to the colony, Chillingworth lives a life apart from others as he feigns being the physician of the minister.  His subjugation of his heart for his intellect furthers his alienation from the community of man in his quest for revenge, turning himself by his own admission into a fiend.  In the end, Chillingworth has alienated himself from his fellow man in his quest for his one victim and from his God.

I cannot give you an exact page number, but there are two chapters you may want to revisit.  The first part that shows his desire to be isolated is in Chapter 3 - in his conversation with Hester about keeping the secret of his identity.  This shows that he wishes to live among these people - but live in a lie.  In so doing, he seals his fate that he never be fully accepted or known.

This proves true in the very last scene of the book (chapter 23) when Dimmesdale approaches the scaffold.  Chillingworth desperately tries to stop him, then admits that this is the one place Dimmesdald could have escaped him.  The most public place in the entire colony is the one place that would have revealed truth - including Chillingworth's identity.

Finally, we see in the last chapter of the book, that once Chillingworth's life purpose (of revenge against Hester and Dimmesdale) is gone, he dies.  His entire reason for life was to be a silent punisher with no identity.  The only person in the entire town he has connected himself to (as he severed the connection with Hester, basically, in chapter 3) was Dimmesdale.  Once Dimmesdale was dead, Chillingworth, in his isolation, dies for lack of purpose.

For Hester, her scarlet letter seems to set her apart in every possible way. She even lives apart from the remainder of her community in a small cottage surrounded by "scrubby trees" that seem as though they exist in order to conceal something. Moreover,

In all her intercourse with society [...] there was nothing that made her feel as if she belonged to it. Every gesture, every word, and even the silence of those with whom she came in contact, implied, and often expressed, that she was banished, and as much alone as if she inhabited another sphere, or communicated with the common nature by other organs and senses than the rest of human kind.

Even when Hester would stand in a crowd, there seemed to exist a sort of magic circle around her, into which no one else would come. She even found that she was sometimes the subject of the Sunday sermon. She became completely alienated in almost every way: religiously, geographically, legally.

As far as Dimmesdale, because his sin is not known, his isolation is of a different, invisible sort. One day, Hester

[...] beheld the minister advancing along the path entirely alone, and leaning on a staff which he had cut by the wayside. He looked haggard and feeble, and betrayed a nerveless despondency in his air, which had never so remarkably characterized him in his walks about the settlement, nor in any other situation where he deemed himself liable to notice.

Thus, Dimmesdale has been isolated by his guilt, and because he has hidden his guilt, he purposely conceals his misery (for it is his guilt that makes him so terribly unhappy), and so becomes emotionally isolated from everyone around him.

For Chillingworth, his single-minded purpose, to exact ultimate revenge on Hester's co-sinner, forces him to become emotionally isolated as well. He must operate in secret, attempting to ferret out precisely what the minister most wants to conceal, and so he has time for little else. The two men even move in together, further isolating them both. Chillingworth's "intimate revenge" requires intense focus and, thus, demands exclusivity.