In order to fully answer the question, let us first assume that "scarlet letter" includes both the symbol of the punishment that Hester bears as well as the piece of cloth she wears. In this way, each of the main characters in the story are differently affected by the scarlet letter.
Hester Prynne: at first it is an embodiment of shame, isolation, and societal disgrace. However, as she accepts her fate, it comes to embody inner strength, empathy, and a burden she believes she deserves and eventually embraces.
Chillingworth: it is a daily reminder that he has been wronged (as Hester's husband) and his shame, though hidden, manifests itself in the form of rage and revenge. The letter, as Hester's punishment, allows him to forgive his once wife, in the belief that she has received what she deserves. But then it causes him to focus his rage and revenge on the other half of the letter: Dimmesdale.
Dimmesdale: because he does not physically wear a badge of shame like Hester, the letter symbolizes internal shame and torment, which may arguably be worse than what Hester must publicly endure. Seeing Hester punished while he hides his guilt drives him mad, makes him physically weak and sick, and the audience is lead to believe that he somehow comes to bear a physical mark of his shame on his chest as a result. He believes wearing the letter (admitting his sin) is the only thing that will free him from inner-torment.
Pearl: as an ironically intelligent and precocious child, the scarlet letter upon her mother's chest is a symbol of hypocrisy and public ignorance. She frequently laughs, plays, or childishly pokes fun at the letter itself, which reveals that her knowledge of society and its flaws go deeper than her age and experience should. In some ways, the letter causes Pearl to judge her mother, not because of what she has done nor her punishment, but for attempting to conform to the rules of a society which Pearl clearly sees are flawed. On the other hand, she does not recognize her mother without the letter, showing that it has come to such an innate part of her existence, that she cannot imagine life in its absence.