Jem knows Scout well. He understands that she sees being a girl as a negative thing. Scout looks around and, with the exception of Miss Maudie, sees women as being weak and restricted by society. She doesn't want to wear dresses and go to tea. She wants to play, like the boys do. Jem is clever older brother -- he knows the best way to convince Scout to do what he wants is to accuse her of being a girl. It's a dare.
Jem is noticing that Scout is becoming more and more hesitant in doing things that involves Boo Radley. I think you might be refering to the part in the novel where the kids are in the Radley garden. As soon as Scout pipes up about her fears, Jem says that Scout is acting like a girl. He sees Scout as nagging him into doing the right thing rather supporting him. Also Dill is with the Finch kids during this episode. Dill does not show any fear in being on the Radley property and he is close in Scout's age. The only reason, Jem reasons, that Scout doesn't want to see Boo is the fact she is a girl.