What do these lines from Hamlet mean?
1. I would have such a fellow whipped for o'erdoing Termagant (II,ii,14)
2. It out herods Herod (III,ii,14)
3. “For O, For O, the hobbyhorse is forgot”
You have asked multiple questions here, although your first and second are connected, since the entire line is "I would have such a fellow whipped for o'erdoing Termagant, it out herods Herod, pray you you avoid it."
This is from Hamlet's advice to the players and he names two characters, Termagant and Herod, that were usually over acted. He is giving advice as to what good acting is and isn't in this speech.
In Hamlet's (and Shakespeare's opinion) too much is too much. Over acting is to be avoided, IE don't "tear a passion to tatters", don't "saw" or "split" the air.
Whether you are a classical actor doing Shakespeare or you are an actor doing a modern piece, this is still good advice. Good acting shouldn't look like acting at all. Chewing the scenery might impress the unenlightened but it shouldn't be mistaken for good acting.