To be honest, you could argue either way.  I would argue that he was ashamed of her because he never appreciated anything about her, and since his life was all about amassing greater possessions and using everything he had, without ever being grateful for the work his family did for him, he felt she was just one more of her possessions.  As such, she wasn't the prettiest or the most perfect so of course he was ashamed of her.

Of course, if it were a test question, I would answer true, he was very clearly ashamed of her and even went so far as to carry on his affairs basically in plain sight.  The way he ridiculed her also gave clues that some of his disdain for her and perhaps the shame he felt because of her was due to her physical impairments.