Well, brown (like gray and black) in The Scarlet Letter is one of the drab colors that the Purtians surround themselves with. It tended to symbolize the severity of the Puritan culture and the fact that little life or joy resided within the towns ruled by Puritan people. For example, in the text, Hawthorne describes the dress of the Puritans to set the mood of the book and the people in it.
The crowd was dressed in sad-colored garments and gray, steeple-crowned hats.
Those colors stand in stark contrast to the vibrant scarlet letter A that Hester is forced to wear, and the gold stitching that she weaves into her garments. It is meant, I think, to show that although Hester was the "sinner", that she had real life in her, while the townspeople did not.