In Shakespeare's Hamlet, the lines you cite literally mean that Hamlet would have an actor whipped for overacting, because overacting creates an overblown performance compared even to an overblown tyrant.
Hamlet is offering acting suggestions or advice to the 1 Player. He tells the actor to not overplay emotionally powerful speeches. He mentions overblown hand gestures and impassioned words presented without restraint as things to avoid. He says not to concentrate on pleasing the groundlings (those patrons in standing positions on the floor of the playhouse who appreciate only mime shows and spectacle). Impassioned speeches must appear natural, and not be overdone just to impress the masses.
Termagant is thought to be a Mohammedan deity, and is represented in medieval mystery plays as a violent and ranting personage, according to the note on the text in the Norton Critical Edition of the play. And Herod, of course, is the cruel tyrant from the New Testament. Hamlet would, then, have an actor whipped for overdoing a violent and ranting Termagent, because doing so would present a character even more cruel and violent and outlandish than Herod's actual "performances."
Thematically, the scene focuses on acting, playing roles, restraint, precision. Hamlet acts with restraint in the play concerning his revenge, firmly establishing Claudius's guilt instead of acting rashly, as Laertes does, for instance, after his father's death. And almost everyone acts and plays different roles in the play: Hamlet pretends to be mad; Ophelia acts like she doesn't love Hamlet and returns his gifts to her while Polonius and Claudius spy on the two of them in order to judge Hamlet's reaction; Ros. and Guil. pretend to be acting only as Hamlet's friends, when they are really acting on behalf of the king.
Hamlet, of course, readers assume, wouldn't really have an actor whipped for overacting. He is just making a point.