I think you have identified a major theme. Isolation is something that, as #5 states, Pearl, Chillingsworth, Dimmesdale and Hester all suffer. It is clear that the illicit affair is a secret truth that isolates Hester and Pearl from the rest of their people, but in the same way it isolates Dimmesdale, as he bears the truth of the secret but cannot share it with anyone. Chillingsworth again is another bearer of a secret that cannot be revealed, which of course isolates him too. And lastly, Pearl and Hester are physically isolated by the location of their house and because of what has happened. Pearl is isolated anyway because of her "elf-child" ways and the fact that she is so different from the other children in this Puritan society.
All four of the major characters are isolated throughout the novel. Hester is isolated because of her sin and the letter she is made to wear. She is able to briefly connect with Dimmesdale in the forest, but that ends quickly enough as Chillingsworth spoils their plans. Although Pearl is her only companion, Hester is isolated from her as well because she cannot comprehend her. Sometimes Pearl is loving and sometimes she is downright cruel, even to Hester. Pearl herself is isolated because the other children mock her, and she shares the ignominy of her mother.
Dimmesdale is emotionally separate from everyone in his life because of his sin and guilt, and Chillingworth's isolation comes first from the fact that he is living a lie (no one knows his real identity or purpose) and second of all from his transformation into a thing of evil as a result of his revenge.
Perhaps the character who is the most isolated and set apart from his fellowman is Roger Chillingworth, who from his youth led a life of isolation as a student. Moreover, as an older man who seeks revenge against the minister, he sacrifices his true identity as well as his very humanity. In contrast, Hester and even Dimmesdale still retain theirs in their interactions with the community. In addition, while Hester and Dimmesdale have committed the one sin of the flesh, Chillingworth has committed two sins, one against Nature, and one grievous sin against the very soul of man and against God Himself. (His very name connotates his separation from any goodness.)
The first sin against Nature occurs when Roger Chillingworth marries Hester knowing that his young and passionate wife does not love him; he knows, too, that he is not a proper husband for her:
Mine was the first wrong , when I betrayed thy budding youth into a false and unnatural relation with my decay (Ch. XV)
Chillingworth's truly atrocious sin begins at the first scaffold scene where he commits what Hawthorne terms the "unpardonable sin," the sin that denies human nature and God. it is the sin of the subordination of the heart to the intellect. Chillingworth becomes willing to sacrifice Dimmesdale to gratify his own selfish interest in vengeance. Of course, this is a sin against God as "Vengeance is mine, saith the Lord" and "Thou shalt not kill" as Dimmesdale's spirit is destroyed by the evil Chillingworth, the physician who who plays God. In fact, from this heinous action, Chillingworth becomes a virtual devil.
As such a devil, Pearl fears him as "the Black Man" while Dimmesdale tells Hester in the forest that he "fears" him. Chillingworth himself admits to Hester: "I have already told thee what I am! A fiend." (ch. XIV) In sacrificing his human sympathies for his desire for revenge, Chillingworth has isolated himself totally; he is separated from all of humanity as well as from God.
Hester is isolated physically since she lives on the edge of town, morally since she has commited adultery, and socially since she is considered an outcast...at least in the first part of the book. Later, she becomes considered a needed part of the community as she provides a necessary service and volunteers. The results of her isolation include a stronger bond with Pearl, and a stronger self esteem. She holds her head up high and does not cower. She does not hide behind her Scarlet "A". Because of this, members of the community come to admire her strength and her heart.
The Reverend Dimmsdale is isolated by the cloth. Being a preacher sets him apart from others. Hester frees him from that profound isolation, but becomes a social outcast as a result of her actions.