Actually, in Shakespeare's Romeo and Juliet, Rosaline is a member of the Capulet family (as mentioned) and, in this case, is Capulet's niece. This is an excellent question.
I don't believe there is a way to know the "correct" answer—or even if there is one—but I would surmise that there are probably several reasons why Romeo's feelings are not the cause of upheaval.
In Act One, scene two, Romeo chances to meet Capulet's servant charged with inviting guests to the party, but he cannot read the list. He asks Romeo for help. Romeo reads the list:
Stay, fellow; I can read.
‘Signior Martino and his wife and daughters;
County Anselme and his beauteous sisters;
The lady widow of Vitruvio;
Signior Placentio and his lovely nieces;
Mercutio and his brother Valentine;(70)
Mine uncle Capulet, his wife, and daughters;
My fair niece Rosaline and Livia...
My first reason is based mostly on conjecture based upon what little we know of Romeo's relationship with Rosaline. It would appear that he simply "moons" after her, wanting her love, which is not returned. There is no way to know that the relationship has brought them closer than writing letters. Either way, there is no danger that anything will become of it as she wants to join the Church. This association may not be considered threatening to the Capulets.
Secondly, the feud exists between the Capulets and Montagues, and we can certainly appreciate how deeply the animosity runs between the two families when the servants are ready to kill each other. However, Romeo presents no threat. He is not at all involved in the fighting, only in "the loving." We also know that at least at the beginning of the play, Capulet has no hard feelings toward Romeo when it is discovered that he is crashing the masquerade party. Capulet is firm in denying Tybalt the chance to confront Romeo at the party. In fact, Capulet finds nothing offensive in the person of Romeo or his uninvited presence at the party, but defends him as a seemingly decent young man, in Act One, scene five:
Uncle, this is a Montague, our foe;
A villain, that is hither come in spite
To scorn at our solemnity this night.
Young Romeo is it?
'Tis he, that villain Romeo.
Content thee, gentle coz, let him alone.
He bears him like a portly gentleman,
And, to say truth, Verona brags of him
To be a virtuous and well-govern'd youth.
I would not for the wealth of all this town
Here in my house do him disparagement.
Therefore be patient, take no note of him.
It is my will; the which if thou respect,
Show a fair presence and put off these frowns,
An ill-beseeming semblance for a feast.
Finally, had Romeo been pursuing Capulet's daughter, rather than a niece (and who knows how involved her parents are in the feud), the situation might have been very different.
(It is interesting to note that, unless there is another Mercutio, Romeo's friend Mercutio—and his brother Valentine—are also on the list of the invited. Mercutio is very good friends with Romeo and everyone is sure to know this, but he is also Prince Escalus' kinsman, and included despite the connection with the Montague family. The separation of the two families may not be as clean cut as one might like while living in the midst of a feud.)
Of course, there is no way to know for certain, but I would expect that this line of reasoning would serve in trying to understand this small contradictory element Shakespeare felt the need to include in his tragedy.