In Chapter 6 of To Kill a Mockingbird, Jem is determined to get his pants back because he does not want to get in any more trouble than he already is in. In the previous chapter, Atticus had instructed Jem and the other children not to go to the Radley house. Jem, along with Scout and Dill, ignore those instructions which lead to Jem losing his pants in the Radley's fence. When a gun sounds, the whole neighborhood gathers in front of the Radley house to gossip about the gunshot they all heard. Apparently, Nathan Radley shot his shotgun in the air to scare off an intruder.

When Atticus interrupts the gossip to ask Jem where his pants are, Jem and Dill hatch an elaborate lie to explain why Jem is not wearing pants. Jem knows that this lie will not hold up come morning when Nathan Radley will likely find Jem's pants, fingering him as the intruder. He is mostly afraid that being caught disobeying his father and then lying to him will greatly disappoint Atticus. We know that Jem holds his father in high regard and wants to live up to his expectations of him. Therefore, he feels it necessary to take the risk of returning for his pants in order to maintain his relationship with Atticus.

You can find the answer to this in Chapter 6.  I would say that Jem wants to get his pants back because he does not want to disappoint his father and make his father think less of him.

Jem tells Scout that Atticus has not whipped him in a very long time.  He says that he wants to keep it that way.  It does not appear to me that he is really afraid of the pain of being whipped.  Instead, it seems that he is proud of the fact that he and his father sort of related to each other as adults.  He does not want his father to think of him as some kid who needs to be whipped.