Why do composers, authors, and screenwriters so frequently use Romeo & Juliet as a basis for their work?
A great question.
There are several answers to this.
I'll start with the cynical one. In formal logic, there's something called "the bandwagon fallacy," or the appeal to popularity. This is the idea that people do things to be part of a larger group, or, as many mothers put it, if your friends all jumped off a bridge, would you do it too? In this case, the bandwagon is the trend for everyone to do and re-do Shakespeare. People do it because a lot of people do it, and they don't want to be left out.
A related but more positive version of this answer is the idea of cultural literacy. People make new versions of Romeo and Juliet because a lot of people know it. It becomes a shared frame of reference, a short hand, a language in itself. It is like a standard of measurement.
Now, to move to even more substantial reasons, look at the richness of Shakespeare in general. In language, themes, and characters, there's a lot in Shakespeare. There are few lines more famous than "O Romeo, Romeo! wherefore art thou Romeo?" and rightly so. That cry sums up a lover who is crying out to understand the forces that keep her and her lover apart.
This play is so rich, it's like there is extra, or excess. Look at some of the characters who aren't Romeo or Juliet, like Mercutio. His "Queen Mab" speech is like frosting, in that it is so rich, so very rich, and not really needed for the core love story. But it is sheer poetry, and makes everything else deeper.
Finally, there is the core dramatic situation. It is young love, but it is also forbidden love. It is a daughter crossing her father, and a young man crossing his entire clan. There have been feuding groups throughout human history, and love (or lust) has always crossed those canyons between groups. That means that you can adapt the play to many periods and cultures. There's a universality here. There's also high drama. The stakes could not be higher. There's social disapproval, respect of family, and eventually, death. That gives composers, directors, and actors a chance at showing ultimate passion.