Jem had to go back for his pants because the lie Dill told to Atticus didn't involve his pants being destroyed, only lost. He said he had lost them in "strip poker." Jem couldn't argue with that lie and come up with a better one where the pants were actually destroyed or else he would risk exposing the lie, so he had to go along with it.

If he hadn't come up with the pants relatively soon, Atticus would have punished him for losing them permanently, a punishment Jem seemed eager to avoid when he said he had not been "whipped" for a long time and he didn't want it to happen again. He clearly has a healthy respect for Atticus and is also afraid of the whip, as he should be. Atticus would have either punished him for losing the pants (something it would cost money to replace) or have punished him for lying, had he found out how the pants were really lost.

So, Jem really had no choice but to go back for his pants, as scary as that prospect was.

Despite being warned by Atticus to leave Boo Radley alone and to stop “tormenting” him, Jem and Dill just can’t let go of their curiosity and obsession with seeing Boo.  When their attempt to poke a note hung on a fishing pole through a window at the Radleys to ask Boo out for ice cream fails, Jem and Dill decide to sneak onto the Radley property and peek in a window on the back porch.  While Dill is a lookout, Jem quietly peers in the window only to see a dark shadow move across the back porch wall.  Jem takes off and gets his pants caught on the Radley fence.  In order to get loose, he takes off his pants and leaves them there.  Nathan Radley (the ominous shadow) fires a shotgun in the air to scare off the intruders.  This brings out the neighborhood to see what happened.  Atticus discovers Jem without pants and asks where they are.  Dill quickly lies and tells Atticus that Jem lost them playing strip poker.

Jem knows he has got to go back and get his pants, or it will be discovered that he went against Atticus’ orders to leave the Radleys alone.  He tells Scout that he doesn’t want to disappoint Atticus and get a whipping for going against Atticus’ wishes.  Jem is afraid of not only getting in trouble, but he also doesn’t want to lose the respect of his father, so he goes back to get his pants and finds them mended and draped over the Radley fence.

Jem and Dill learn their lesson through this scary encounter with Nathan Radley, and their curiosity wanes because of Atticus’ insistence to respect the privacy of Boo.

Jem lost his pants on the Radley fence by making a raid on the property in the hope of catching a glimpse of Boo from the back window...