Why does Tybalt hate the Montagues so much in act 1 of Romeo and Juliet?

The short answer is that we are never told. The Prologue to the play that the "two households" of Capulet and Montague have an "ancient grudge" but Shakespeare never explains what that grudge is or why it started. It may be that Tybalt as a Capulet is simply continuing the grudge for its own sake with no other motive. Certainly in the Masque scene, he accuses Romeo of have attended to "fleer and scorn" at his family and vows to kill him "by the stock and honour of my kin". This would imply that Tybalt's anger is directed at some perceived slight to his family honour.

In Act 3 scene 1 Tybalt tells Romeo that he cannot excuse the "injuries" done to him, which may be a reference back to the masque or some other undisclosed incident.

Another answer is more psychological and character-driven. In Act 1 Tybalt tells Benvolio that he hates the word peace "As I hate hell, all Montagues and thee" and certain productions have presented Tybalt as an almost psychotic character who enjoys and revels in the violence and aggression of the grudge. Baz Luhrman suggests this in his film version.

Why does Mercutio hate Tybalt in act 3, scene 1 of Romeo and Juliet?

In Shakespeare's Romeo and Juliet, in Act III, scene i, I think that Mercutio is something of a show-off, and has a lot to say to mock others, even Benvolio. I don't think that his intent is malicious: he is just someone who doesn't take too much seriously and loves to stir things up a little. However, when Tybalt comes along, a member of a noble house, and is rude, Mercutio takes great offense. He is not someone to be easily pushed around. He also get annoyed when Romeo will not engage in trading insults with Tybalt. (Of course, Mercuitio does not know that Romeo is now married to a Capulet, but thinks Romeo is simply acting cowardly.)

Returning Tybalt's disrespect, Mercutio insults him:

Tybalt, you rat-catcher, will you walk? [step aside]...

And then when Tybalt asks what Mercutio wants with him, Mercutio responds with another insult or two:

Good King of Cats, nothing but one of your nine lives that I mean to make bold withal, and as you shall use me hereafter, dry-beat the rest of the eight. Will you pluck your sword out of his pilcher by the ears? Make haste, lest mine be about your ears ere it be out.

Mercutio has already called Tybalt a "rat-catcher," and now he carries on the comparison stating he wishes to take one of this cat's proverbial "nine lives," and if Tybalt continues to insult Mercutio, Mercutio will beat up the remaining eight. Then he challenges Tybalt to draw his sword: quickly or Mercutio will cut off his ears. Tybalt draws and they commence to fight.

Tybalt is a hot-head, and he hates the Montagues with a passion: not because he agrees with their dispute—as far as we know, neither Capulet or Montague remember what this age-old feud is about. But Tybalt needs little reason for stirring things up; he is a trouble-maker. And Mercutio, despite his biting wit and over-the-top sense of humor, has no time for this blackguard (scoundrel). He doesn't have much time for the Capulets, either, but for him it's more personal: based on how Tybalt directly treats Mercutio simply because he is friends with Romeo.

It is under these circumstances that Tybalt reaches around Romeo's body, as he tries to stop the fight, to take a cheap shot at Mercutio that eventually brings about the young man's death—as he is dying, Tybalt has run away.