There is a lot of significance attached to the scarlet letter. Initially, Hester is forced to wear it as punishment for committing adultery, but throughout the novel, the letter almost takes on a life of its own, and its significance changes. The trial of wearing the letter over a period of time strips Hester of her womanly features. In chapter 18, Hester removes the letter for a brief period of time, and when she does so, the narrator tells us that her womanly beauty returns. When she puts the letter back on, she loses that beauty.
Because of Hester's actions throughout the novel, the letter also changes in meaning. Initially, it stands for adultery, but during the course of the novel, Hester helps so many people in a quiet and humble manner that the townspeople begin to say that the letter means 'Able'.
The letter serves as Hester's punishment, but even when she is allowed to remove it she refuses, because it has become a part of her. It has humbled her, punished her, and changed her.