In "Desiree's Baby," Armand was eight years old when his mother died. How could he not remember her being black?
Armand did not know that his mother was black because she was "passing" as a white person. That is, she was of mixed race herself and was very light skinned. In early Louisiana and most of the south, there were laws preventing mixed-race marriages. People were so adamant that there be no mixing of races that they invented terms to describe the proportion of the mixture.
For instance, in the story, Desiree is watching two quadroon boys fan her sleeping baby son. "Quadroon" was the label placed on persons of one-fourth African ancestry. One of the parents of a quadroon would have been called a mulatto--or biracial. Other designations include quintroon (one-fifth African ancestry) and octoroon (one-eighth African ancestry).
Armand's mother may have been no more than one-eighth African American and displayed no African physical characteristics at all. So it is very plausible that the boy Armand would never have imagined that mother wasn't considered white by the law.
By the way, I am one-eighth Cherokee, and I have blond hair and blue eyes and very pale skin.