At the beginning of chapter five, Scout is lamenting the fact that all Dill seems to want to do is to hang around Jem. She says of Dill and Jem,
"they spent days together in the treehouse plotting and planning, calling me only when they needed a third party."
Because of this, Scout is a bit resentful, and spends more and more time "with Miss Maudie Atkinson on her front porch." Also, because Scout is a bit jealous, it makes her accept the challenge that Jem issues in regards to standing watch while Dill and Jem try to deliver a note to Boo. She didn't want to; she thought it was a super bad idea, and had felt bad for tormenting Boo. In fact, she was trying to get Jem to lay off. But, because she was feeling left out, she agrees to this plan. Jem ends up getting into trouble for the escapade.
So, Jem spends most of his time with Dill that summer. I hope that helps!