Here's what Romeo says:
I do protest I never injur'd thee,
but love thee better than thou canst devise.
And so, dear Capulet, whose name I tender
As dearly as my own, be satisfied.
Mercutio is the next to speak:
O calm, dishonourable, vile submission.
Alla stoccata carries it away.
Tybalt, you ratcatcher, will you walk?
You're right to assert as you do that Mercutio does not know that Romeo's love for the Capulet's name is anything to do with Juliet: in fact, he dies never knowing about Romeo and Juliet's relationship.
Mercutio simply thinks that Romeo's refusal to fight is because of cowardice, and because he is scared of Tybalt: and his submission to Tybalt is "calm, dishonourable, vile". As his friend won't fight Tybalt, he draws his sword to fight him himself. And the rest, as they say, is history.
In this scene of "Romeo and Juliet," Romeo comes upon Tybalt and Mercutio who are verbally challenging each other. Tybalt accuses Mercutio, "thous consort'st with Romeo"(33). Perceiving the tension between the two men, Romeo seeks to ameliorate the situation by telling Tybalt that he now loves him and is not a villain. (Romeo has married Juliet now and is part of the family.)However, Tybalt does not accept Romeo's love as an excuse for injuries done to the Capulets. Romeo counters that he has never injured him, "But love thee better than thou canst devise"(57).
With this second profession of love by Romeo, Mercutio is disgusted: "Oh, calm, dishonorable, vile submission!(62) he exclaims. Mercutio is greatly disturbed by what he perceives as a servile attempt by Romeo to ingratiate himself to Tybalt. The dramatic irony here is not lost on the reader who knows that Romeo is not cowering to Tybalt, but is attempting to tell the man that he is part of the family. However, neither Tybalt nor Mercutio understand. Mercution decides to quiet Romeo's "effiminacy"--as later Romeo exclaims--and defend the honor of the Montague family.
Romeo's tragic mistake of telling Tybalt he "loves" him has escalted into a duel in which Mercutio is slain.