To be grotesque is to be unnaturally strange or "bizarre" (Random House Dictionary). It is also to be ridiculously incongruous, which is also to be out of keeping or "inappropriate" (Random House Dictionary). In other words, one who reflects the "principles of the grotesque" has a grotesque character by being completely inconsistent with respect to the rest of society. In other words, one could consider a grotesque person to be completely uncivilized.
There are many ways in which Sir Toby demonstrates being grotesque. For one thing, Shakespeare uses his character to embody the themes of merriment, foolery, and wild behavior that were characteristic of the holiday celebrated on the twelfth day after Christmas, called Epiphany, for which the play is named for. As a result, Toby is characterized as being quite wild and even bawdy, meaning obscene. One example of his wild, bawdy behavior is his consistent state of drunkenness, which also represents the drunken behavior characteristic on Epiphany. We first learn of Toby's drunken behavior in the very third scene when Maria scolds him for his wild behavior, warning him that Olivia is very offended by his habit of coming home drunk so very late at night. She even warns, "[Y]ou must confine yourself within the modest limits of order," showing us just how incongruous Toby's behavior is with the rest of civilized society's behavior (I.iii.7-8).
In addition to his continual drunken state, he also demonstrates being grotesque by being what other critics have called a "freeloader" (eNotes, "Sir Toby Belch (Character Analysis)"). He demonstrates freeloading by living off of Olivia, plus taking advantage of his friend Sir Andrew. He has brought Andrew into Olivia's house in the hopes that she'll accept Andrew as a suitor. However, what Toby is really interested in is Andrew's money. As long as Andrew is around, Toby has free access to plenty of money. But by the end of the play, Andrew has failed to court Olivia and is out of money, so Toby says exactly what he thinks of his so-called friend. After being injured by Sebastian, Andrew very warmly invites Toby to come along with him into the house so that they can be bandaged, or "dressed together," but Toby's response is, "Will you help an ass-head, and a coxcomb, and a knave? a thin-faced knave, a gull?," all of which are adjectives that apply to Andrew, showing us just exactly what Toby really thought of Andrew all along (V.i.212-13). Furthermore, his sudden use of these adjectives show us just how much he was really using Sir Andrew to take advantage of him, which is also incongruous with the rest of society, proving that he is grotesque.