I have to agree with many of the above posters that the issue for physics in particular is the math. While motivation for any subject at any level is an issue for all educators, I think the importance of math and the difficulty of the level of math used in physics is a huge issue.
Many, many students struggle with math and in addition math anxiety is rampant among students. And while I would like to think that both genders are treated equally the reality is that girls are not encourage in math and science the way they should be.
Physics is an area where you must have a firm understanding of mathematical concepts. Many other content areas don't prove as difficult because they do not require as great an understanding not only of the subject you are teaching but the application of another subject.