First, probably because of the gender stereotyping that lands boys in maths and sciences, while girls head more for the softer subjects, like literature and history. (There are lots of theories for that, but the one I agree with most is that girls are more interested in people's inner motivations (generally speaking) while guys typically like to figure out the outside world. It's certainly not the only, nor probably the best theory, but it's what I see in my own experience.) Physics, unfortunately, has a reputation for being all advanced math and formulas. So if the stereotype is keeping girls out of higher-level math classes, they will probably not do well in physics.
Personally, I think we also kill the interest in physics for everyone by trying to instill the basics before learning what the basics are for. I hated physics (and I am a "girl") i high school because I was bad at math. But in college, I had to take a Physics for Stagecraft course because I was working on backstage crew in the rigging and lights. Suddenly all that boring stuff had application. Because of that class, I see the basics of physics everywhere, and it's beautiful.
If it has no application, it's just a bunch of boring math formulas. Which makes it hard for many girls who just don't, for whatever reason, like math.