At the beginning of the story, Hester is the outcast of the town. They revile her because of her sin, judging her and alienating her. She lives on the edge of town and is not accepted into mainstream society.
As the years pass, the town will slowly start to accept and then to honor Hester. It begins with her sewing. Although they condemn her, the ladies of the town hypocritically accept the work of Hester the seamstress. As more time passes, the actual ridicule fades away. Hester's charity work is gratefully accepted. The town begins to see her as an "elder" in the traditional sense, meaning that she is applied to for comfort and advise. When she is buried, her grave is marked with an "A", but that "A" is no longer connected to the idea of adultery. It has become connected with angel and with able, the true nature of Hester.
What is ironic about her relationship with the town is that, as bad as it is in the beginning, Hester chose it herself. She was not required to stay, but made staying her penitence for her sin:
"Here, she said to herself, had been the scene of her guilt, and here should be the scene of her earthly punishment; and so, perchance, the torture of her daily shame would at length purge her soul."