Tybalt is part of the younger generation of Capulets to continue the feud which has gone on for years and years.  Incited by the opening street confrontation between the two families, Tybalt is further angered when he learns that Romeo and friends have come uninvited for his close cousin, Juliet.

Now, in Act III Tybalt wants to know why Romeo was at the party and ascertain his motives, so he calls out in a taunting manner to Mercutio

Mercutio, thou consort'st with Romeo--

When Romeo does arrive, Tybalt insults him,

Romeo, the hate I bear thee can afford

No better term than this--thou art a villain. (III,i,50)

Tyblat further tells Romeo that he has caused him "injuries," and challenges him to a duel. For, the effrontery that Romeo and his friends have displayed by coming to their enemy's house uninvited is considered a grave insult to the Capulet name.  In the Italy of the fourteenth century, family name and its honor were of paramount importance.

Tybalt is indeed upset with Romeo for appearing at the party. He feels Romeo has insulted his family, and owes him a duel to restore the Capulets' honor:

Boy, this shall not excuse the injuries
That thou hast done me; therefore turn and draw.

Tybalt is not known for his calm, even-headed personality, as witnessed in the party scene. Lord Capulet essentially has to hold him back, in order to avoid a fight in the house. Tybalt has let his grudge against Romeo simmer, and he is even more angry after what he considers an insult from his uncle at the party.

As for what he hopes to accomplish, he wants to save face, and prove himself the better swordsman of the two. However, he fights Mercutio instead, but does get his duel with Romeo eventually.