The quote in question can be found at the beginning of Act Two, Scene 3 when Hamlet is critiquing the performances of the Players before their scene. Hamlet explains to the Players that he hates when actors exaggerate their lines and use excessive hand gestures. Hamlet also mentions that he gets annoyed whenever actors overpower their voices when they are supposed to portray emotion. Hamlet then tells the Players:
"I would have such a fellow whipped for o'erdoing Termagant. It out-Herods Herod. Pray you, avoid it" (Shakespeare, 3.2.12-14).
Hamlet is alluding to King Herod of the Bible, who was the Rome-appointed king of Judea when Jesus was born. Throughout the Bible, Herod is portrayed as a ruthless tyrant. After speaking to the wise men concerning the birth of the King of the Jews, Jesus, Herod orders his soldiers to murder any male child two years old or younger in Bethlehem and the surrounding area. In context, Shakespeare is employing a hyperbole when he says, "It out-Herods Herod." Essentially, Hamlet means that whenever performers excessively exaggerate their lines, they have the ability to inflate the wickedness of the ultimate villain, which in a Christian society would be Herod.
Your original question had to be edited because it actually contained multiple questions. Please remember that enotes only allows you to ask one question at a time.
The identified quote comes in Act III scene 2, and is actually one of the parts of this play that adds humour to the unyielding bleakness and despair that threatens at times to dominate. Hamlet, in offering advice to the Players about how they should act and deliver lines, actually is used by Shakespeare as a humorous critique of bad acting in his time. In particular, the quote identified is an example of hyperbole, as Hamlet exaggerates how terrible bad acting is:
O it offends me to the soul, to hear a robustious periwig-pated fellow tear a passion to tatters, to very rags, to split the ears of the groundlings... it out-herods Herod.
Herod was of course the tyrant who ruled Palestine and ordered all babies in Bethlehem to be slaughtered because of the prophecy that was made about the birth of Jesus as the King of Kings. Thus to "out-herod Herod" is to literally out-do in wickedness or violence. The way in which Hamlet applies such a description to a bad actor who can't deliver his lines properly indicates the extent of the hyperbole.