Why does Shakespeare give Malvolio an ambiguous ending in Twelfth Night?
I am glad you have identified that the ending of this excellent play is not entirely the "happy ending" that we would expect of a Shakesperian comedy. It is vitally important to consider how the character of Malvolio and what happens to him fits in overall to this "comedy." Certainly there is a sense in which this play could be regarded as a "tragedy," and definitely we are left with the feeling that Malvolio's punishment at the hands of Maria and Sir Toby does not fit his crime of self-importance and arrogance. His final words, which he utters before leaving, strike a discordant note in the otherwise happy ending:
I'll be revenged on the whole pack of you!
Certainly this ending makes us ask some hard questions about the play. The festival of Twelfth Night was a riotous affair, characterised by disguises and characters being free to operate outside of their social circles. It was the last blast before the Christmas season was over and the long, hard and dark months of January set in. Of course, this party spirit is best symbolised by the character of Sir Toby Belch, whose die-hard party spirit is reflected throughout the play. Yet, in the final act, the confusion of Twelfth Night is resolved and we are left with the union of the main characters. But to what extent are we convinced with the happy ending? Olivia and Sebastian hardly know each other, and the relationship of Viola and Orsino has been based on mistaken identity. To what extent will their future be happy?
Thus one way of viewing the problematic ending of Malvolio is to see it as a kind of prompt to force us to ask other questions about the supposedly "happy" ending of the play. Maybe it leads us to view the resolution as not being that "happy" after all, as Twelfth Night is over and real life must be returned to.