First of all, thank you for introducing me to this interesting poem and the poet who wrote it, Natasha Tretheway; I must admit that I'd never heard of her before.
Natasha Tretheway was born in Mississippi in 1966 to a mixed-race couple: her mother was African-American, her father was a white man from Canada. In 1966 Mississippi, such a marriage was not only scandalous, but illegal.
Natasha Tretheway was born very light-skinned, but her mother insisted that she identify herself as "black." The poem is about the lies that Natasha sometimes told in order to be accepted by white society.
This is where the poem's colors come in. Natasha would tell people that she was "light-bright near white," but these were just "white lies." This seems to have a double meaning: these lies were harmless (at least in Natasha's mind) and they were about being white and intended for white people.
Amongst black people ("in a black place"), Natasha would say that she was "high yellow, red-boned." These are terms that African-Americans sometimes use to describe their fellows who are relatively light-skinned.
Natasha's mother did not approve of her lying. She would punish her daughter by washing out her mouth with ivory soap. Natasha ends the poem by saying, "I swallowed suds/ thinking they'd work/ from the inside out." The poet seems to be saying that denying her identity was a spiritual "sin" that had to be cleansed "from the inside out."