This question really needs to be on the discussion board to get a wide variety of responses.
Your question asks for an opinion rather than a factual answer, so I'll give you mine. No, I did not get the impression that Armand thought Desiree had been unfaithful to him. I don't think he would have been nearly as hateful to her if she had been. Infidelity could be easily forgiven. Having a parent who was from the "wrong" race could not. When it became obvious to Desiree that her baby had a mixed heritage, she was horrified and immediately blamed herself. After all, she was a foundling, and no one knew who her parents or grandparents were. Armand's family, on the other hand, were important, longstanding members of the community. No way could he be the one to "blame." Very convenient for Armand!