This action occurs in act 2, scenes 3 and 5. The meaning of “MOAI” is one of the hotly debated subjects in Shakespearean scholarship. There are three aspects to your question. One is, why would Maria want to fool Malvolio at all? The second is, why does she use a riddle? And third, what does that riddle mean?
Maria becomes annoyed at Malvolio for his holier-than-thou, puritanical attitude, which broke up the party with Sir Andrew and Sir Toby. She decides to have fun at Malvolio’s expense to “make him a common recreation.” She correctly sizes up his main character flaw, his vanity, saying that he is
[...] an affectioned ass,
that cons state without book and utters it by great
swarths: the best persuaded of himself, so
crammed, as he thinks, with excellencies, that it is
his grounds of faith that all that look on him love
him; and on that vice in him will my revenge find
notable cause to work.
Maria deploys a riddle in service of her scheme because she expects it will occupy Malvolio’s interest but he will not struggle to decipher it. Instead, she correctly predicts, he insists that he knows at once what it means. Calling the solution plain as “daylight,” he says, “I do not now fool myself” by overthinking it—but of course fooling himself is exactly what he does. Sir Toby and the others are aware of his mistakes, pointing out that Malvolio does not look around him to notice that he is being mocked. The great enjoyment they get out of seeing him make a fool of himself add spice to what Fabian calls Maria’s “dish of poison.”
“M.O.A.I” has been called the “greatest riddle” in all of Shakespeare, notorious for its difficulty to interpret. In the play itself, various characters comment on the meaning and on the challenge it presents to Malvolio. The most common explanation is that it refers to Olivia, if the letters are rearranged to I. A.M. O, “I am Olivia.” Another, as Malvolio himself quickly realizes, is that the letters are all in his name. Many interpreters assume it is an anagram, while others have interpreted the letters in Latin. Still others have run through multiple interpretations, only to conclude that it is deliberately meaningless, absurd—a trick that Shakespeare has played on all of us.
Hassel, R. Chris, Jr. 1974. “The Riddle in Twelfth Night Simplified.” Shakespeare Quarterly, 25(3).