As Jem approaches adolescence, he becomes quieter and more easily agitated. He is not as willing to spend time with his little sister and her silliness. His patience doesn't last as long as it used to with her. He spends much more time alone, reading, or walking off into the woods.
This happens about the time that Tom's trial begins because Jem is having trouble understanding all that is happening in terms of race and the differences in the people. He is genuinely shocked that the verdict comes back as "Guilty" for Tom Robinson.
Scout does not seem to fully comprehend all that this means, but she does get that Jem isn't spending as much time with her and it upsets her. Jem is distancing himself not because of a lack of love, but because he's trying to figure out where he belongs in the big picture.