At the beginning of chapter 1, entitled "The Prison Door," the narrator says,
The founders of a new colony, whatever Utopia of human virtue and happiness they might originally project, have invariably recognized it among their earliest practical necessities to allot a portion of the virgin soil as a cemetery, and another portion as the site of a prison.
Thus, the Puritans built a cemetery and a prison because they knew, no matter how much they hoped it could be otherwise, that people would inevitably do two things: they would die and they would commit crimes. Just as a cemetery can be symbolic of death, the prison is symbolic of our criminal (and, for the Puritans, sinful) natures; since sins were crimes to them, the prison becomes a symbol of sin.
The imaginary circle that constantly surrounds Hester and Pearl when they are near other people, the circle into which others typically do not venture, could be seen as symbolic of their reputation as sinful beings. It is a physical representation of the feelings many people experience: they want to keep their distance from Hester and her daughter, as though their sinfulness might infect others who come near to them. The narrator says,
Mother and daughter stood together in the same circle of seclusion from human society . . .
Because of their sinful reputations, they are physically set apart.
The character of Chillingworth, himself, could be considered a symbol of revenge. His obsession with taking his revenge on Hester's co-sinner overwhelms him, and he changes from someone who was once good to a person that others associate with the devil. The narrator says,
At first, his expression had been calm, meditative, scholar-like. Now, there was something ugly and evil in his face, which they had not previously noticed, and which grew still the more obvious to sight, the oftener they looked upon him. According to the vulgar idea, the fire in his laboratory had been brought from the lower regions, and was fed with infernal fuel; and so, as might be expected, his visage was getting sooty with the smoke.
Chillingworth's physical and moral declension shows how revenge ravages a person, stripping them of their humanity.