Hawthorne's use of color in The Scarlet Letter is certainly meant to be symbolic, especially when you look at his use of color contrast.
The two colors most often referenced throughout the novel are red and black, and white (often portrayed as "light") is used in contrast to these. First, consider the use of the color red, or scarlet. Hester is sentenced to the punishment of wearing a red letter 'A' upon her chest forever. In contrast to the dull tans and browns that most Puritans would have been wearing, the color red would stick out as something shamefully bright and ironically beautiful. Red here could symbolize a burning fire (of shame or of hell) as this letter causes Hester to feel physically and emotionally punished. Pearl is also described as looking very similar to the scarlet letter that Hester must wear. In this way Pearl and the letter on Hester's chest come to represent the same shame and the same ironic beauty of one another.
The color black is most often used in conjunction with description of Chillingworth (the "black man") and description of the forest. Here, black directly symbolizes evil. Chillingworth, as a character, is set on revenge. His heart does not harbor forgiveness, but is instead darkened by evil and sin. Also, as a doctor, he combines western medicine with the "dark arts" of the natives. His physical and psychological "remedies" for Dimmesdale's sickness only bring more weakness. The black symbolism used for Chillingworth is directly contrasted by Dimmesdale's pale countenance.
Further, the forest is described as dark and full of shadows. The forest itself is meant to represent something outside of the Puritan society, and is therefore considered evil. This is where Mistress Hibbins is said to conduct her witchcraft. This is also where Dimmesdale and Hester meet for the first time away from the community. In the scenes in the forest, the shadows follow Hester and the sunlight follows Pearl, contrasting the sin and shame of the mother with the purity of the child. Here, red, black, and white come together for the first time, reflecting change in each character. Hester casts off her red letter (momentarily) and feels a sense of lightness (and sunlight) on her for the first time in years. Dimmesdale's health is momentarily restored by his connection to Hester, and his pale cheek gains a bit of a blush (red). Throughout this scene, the sunlight through the trees causes the shadows to move about but always seems to remain on Pearl. This creates the sense of movement and change, but suggests that Pearl is a constant, and is pure.