Why is the play a tragedy for Malvolio?
I'm not sure that this play does constitute a tragedy for Malvolio, for if it were, he should come to some awareness and new understanding. Also, others should suffer or experience some negative consequence as a result of his actions, and none do.
Instead, he feels wrongly treated (he was) but does not seem to see how his own character traits set him up for this abuse.
He certainly has the right to feel sad, for Malvolio has most definitely been "notoriously abused"; even the audience giggles at his expense. He has been held up for ridicule, his personality is exaggerated (thanks to the ruse), and he is mocked to within an inch of his sanity. Beneath the humour, however, is an individual whose personality is such that he is found to be ridiculous, and that, for anyone, would be tragic.
I would say that it is because Malvolio never rises beyond being "an affectioned ass."
Malvolio is Olivia's steward, and fancies himself in love with her. It is hard to see him as not getting what he deserves, however, as his arrogance is over-the-top. For example, he has been so snooty to Sir Toby and Olivia's servants and her chamberwoman, Maria, that they plot to get Malvolio in trouble with Olivia. They place a forged letter which seems to be in Olivia's handwriting in which she professes love for someone, begging the receipent of the letter that if he responds, to "be the opposite with a kinsman, surly with servants," to wear yellow stockings cross-gartered and to wear a constant smile.
Of course, Malvolio falls for it. And of course, he never gets the fair Olivia.
To call Malvolio a tragic figure would be a stretch, I agree. On the other hand, to use the classical model, he is a basically good person whose character flaw (in this case, his vanity) brings about his downfall. And though it is easy to ridicule him, we cannot but feel pity for his misfortune. True, he never seems to learn much from his downfall, and he never has a moment of realization of his folly. But Shakespeare manipulates our sympathies so that we see a grain of compassion for him.
It's worth noting that in many of Shakespeare's comedies, there is one character for whom there is no resolution, no happy ending: Jacques, Shylock, Antonio (in The Tempest), and Don John (Much Ado), for instance.
The play is a tragedy for Malvolio because in the play, Malvolio was tricked to believe that Olivia loves him; he thought that he was so smart that Olivia loves him and so at the end he is disheartened by knowing that she loves Sebastian and that it was a trick played by Maria and Sir Toby.