The Montagues are caught at the ball--at least Romeo gets discovered, that is. Tybalt notices Romeo and tells his uncle that he has crashed the party. Lord Capulet tells Tybalt to leave him alone, that all of Verona praises him for being a "virtuous and well-mannered youth." In other words, he's telling Tybalt to enjoy the party and ignore Romeo. He must have known that Romeo would not have come there alone; with all the Capulet enemies around, it would have been too dangerous for him. So even though he knows that there are Montagues in his house, Lord Capulet does not want to disrupt his party and ruin his guests' good time by causing a scene and kicking them out.

If the Montagues are caught, a fight is likely to break out. Now the Prince threatened both families that if they ever disturbed the quiet of the streets again, they would pay for it with their lives. So whether Tybalt killed them or the law of the Prince did, their lives are completely at stake. But, the Capulet's mansion would be private property. So, they are likely to not need have any formal royal intervening. The problem would be an altercation between the two families and the aftermath likely to be investigated by the Prince.

To me, this sort of depends on who catches them.  After all, Tybalt does actually know that Romeo is at the ball and that he is a Montague.

When Tybalt sees that Romeo is at the ball, he goes to Lord Capulet.  He wants to just go and kill Romeo for daring to come to their party.  He says

Now, by the stock and honour of my kin,(60)
To strike him dead I hold it not a sin.

But luckily he is not in charge -- he is not the head of the Capulet family.  It is Lord Capulet who is head of the family and he does not mind having Romeo there.  He tells Tybalt to chill and says

I would not for the wealth of all this town
Here in my house do him disparagement.
Therefore be patient, take no note of him.

So it seems that nothing will happen to Montagues if they are caught.  Except if Tybalt had his way they'd be killed.