The duke doesn't really know Olivia; though he is madly infatuated with her, he hardly speaks to her, and certainly doesn't share any of his deep thoughts with her. In this respect, he does appear to be more in love with love than he is with the "object" of his affections. With Viola, however, the duke reveals his soul; they speak of important issues and spend many a (he thinks) manly hour in intimate conversation. His connection with Viola is deep, not superficial; he loves her.
To hear Orinio speak about them to Viola, he doesn't even seem to *like* women much; certainly he thinks them incapable of the depth of feeling a man such as he can have. And yet, he seems to be willing to acknowledge that as far as constancy goes, women have men beat.According to the duke, men are given to "giddy", "unfirm" fancies, which is to say that they can't be trusted to love the same woman for long. This may not explain his swift change of heart, but it does set the audience up to expect that a man would do so:
however we do praise ourselves,
Our fancies are more giddy and unfirm,
More longing, wavering, sooner lost and won,
Than women's are.
Duke Orsino is very fickle-minded, who is buffeted by strong emotions. When he founds a new breed, he drops Olivia and head for greener pastures, which means switching his care and affection towars another person, which is actually a woman, Viola disguised as a man. It seems weird and confusing at first because at first, he had been pursing Olivia and been praising her, but now, he had headed for Viola and he had a drastic behaviour change that was never seen before when he was in Olivia: diminishing his self-involvement and making him more likable.