Madame Valmonde may always have been worried about Desiree's baby's racial makeup since she had adopted her daughter without any knowledge of her origin. Since it was not uncommon for children who had no home to be of a mixed background; Madame Valmonde may have been concerned upon first seeing Desiree, who had been found sleeping in the shadow of a great pillar. In fact, when Armaud Aubigny asks to marry Desiree,
Monsieur Valmondé grew practical and wanted things well considered: that is, the girl's obscure origin.
While Desiree and Arnaud do not consider that the baby can be anything but what they themselves presumably are, they do not wonder of the child's origin. However, this concern has always been in the minds of M. and Mme. Valmonde.
Have you ever had someone who you had not seen in a while tell you how much you have changed? Physical changes take place over long periods of time. They are not instantaneous. Therefore, since Madame Valmonde had not seen Desiree or the baby for a while, she would have noticed changes in the child more easily than Desiree or Armand.
Desiree and Armand would not have noticed slight darkening of the skin of the baby given they are with the child every day.
I agree with the above post, and I also think that they don't notice because they aren't looking for it -- especially Desiree. Armand may have been hoping that the baby was light skinned like him and would "pass," but when it becomes apparent that the baby is even a small percentage African American, Armand confirms the truth about himself -- a truth he doesn't want to face, so he sends away the evidence of his heritage by blaming it on Desiree's genes.