The scarlet letter that Hester wears on her chest is the very symbol upon which the novel is based. It represents not only the word "adultery" but her public shame in having conceived a child outside of wedlock in a Puritan society. She must wear the letter on her plain clothing daily and show the entire society that she committed a grave sin and is now paying the price. Beyond that the letter is a crimson red color which, later in the novel, comes to correspond with Dimmesdale's own self-inflicted bloody scarlet letter on his bare skin. Though no one but Hester, and then Chillingworth, know the truth, Dimmesdale still punishes himself because he cannot bear the weight of the guilt having to watch Hester suffer alone while he continues to live a relatively normal life without loss of respect from the town.
The Puritans were known to dress in usually dark and dull colors. The red color of the letter would definitely stand out in the town as Hester walked through. Moreover, the color also represents the passion, sin, and guilt that earned her the letter in the first place.