One reason why Olivia falls in love with Viola disguised as Cesario is because Viola is extremely direct and forthright with her, which can be very attractive. Unlike Orsino, Viola is able to view Olivia through critical judgement, which is far more honest than mere blind love. We especially see Viola being openly critical of Olivia in the very first scene in which they meet. For one thing, Viola very boldly declares that what is Olivia's right to give, such as her hand in marriage, is not also her right to withhold, which is essentially an open critique of Olivia's character, calling her selfish (I.v.176). She even further calls Olivia "the cruell'st she alive" for leaving "no copy" of her face, meaning for leaving no offspring (225-27). She even very rightly proclaims Olivia to be "too proud" (234). While one would think Olivia would shun such criticisms, she finds them very attractive because they are honest. Unlike Orsino who blindly sees Olivia the way he wants to see her, Viola as Cesario is seeing Olivia as she really is. Hence, Olivia is attracted to Viola's forthright honesty and her realistic perception. In addition, it's part of human nature to want more of what is unlikely for us to have. Hence, Olivia sees that Viola has criticized and spurned Olivia's character, making Olivia want to please Viola and feel attracted to Viola as Cesario.
Literary critics also point out that another reason why Olivia falls in love with Viola is that she sees they are intellectual and free-spirited equals (eNotes, Forbes, "Viola and Olivia"). Viola first demonstrates both her intellect and her free-spirited nature when she refuses to be sent away at the gate. She even goes so far as to say she is already aware of all the excuses Olivia will give not to see her, but still she will insist on speaking with her. By the time Viola perseveres and gains admittance at the gate through her cleverness, Olivia is already very intrigued by her. Then Viola shows even more free spirit by openly criticizing Olivia to her face. Literary critics point out that both Olivia and Viola are very similar in both their losses and their perseverance (eNotes, Parker, "Viola and Olivia"). Both women have just lost all family members and are left to manage their estates on their own. Olivia, through her own determination, has managed to handle her estate very well and even managed not to succumb to marriage; hence, when Viola enters her life and shows the same amount of perseverance and determination, she becomes very intrigued.