Yours is certainly a legitimate question. The Duke tries to explain why he didn't make his displeasure known in the following lines from the poem:
Who'd stoop to blame
This sort of trifling? Even had you skill
In speech--(which I have not)--to make your will
Quite clear to such an one, and say, "Just this
Or that in you disgusts me; here you miss,
Or there exceed the mark"--and if she let
Herself be lessoned so, nor plainly set
Her wits to yours, forsooth, and made excuse,
--E'en then would be some stooping; and I choose
Never to stoop.
For one thing, he doesn't seem quite sure what it is that displeases him about his young wife. We as readers can't be sure either, since she seems like such a beautiful and charming girl from everything he says about her. Furthermore, he admits that he has no "skill in speech." Even if he knew what displeased him, he wouldn't be able to explain it to her; and if he tried to explain as best he could, he was afraid she wouldn't understand and might even contradict him.
Note that his lack of skill in speech is suggested in the lame rhymes of the open couplets used throughout the poem, e. g., "excuse" and "choose." The rhymes are so awkward and so staggered that a person could read the entire poem without realizing that there are rhymes at all. But they deserve special attention as an important technical aspect of Browning's dramatic monologue.
The Duke is obviously an exceptionally proud man. He is so consumed with admiration of himself that he can't see anybody else or think about anybody else. He doesn't even realize what a bad impression his whole monologue is making on his visitor. He is so proud that he chose not to "stoop" to attempt to admonish his young wife, who might only have been about fourteen years old. (See the portrait in the Wikipedia article referenced below.) This tyrant who considers himself a connoisseur of art is spiritually blind. It would seem that what really disgusted him about his last duchess, the hapless victim of an arranged marriage, was that she was an infinitely better human being than he is.