At the end ofchapter 4 Scout says there were two reasons she wanted to quit playing the "Radley game." One was because Atticus showed up and caught them. Jem doesn't think that Atticus knew what they were doing, but in chapter 5 we find out that Atticus did know.
In chapter 5 Atticus catches them trying to give Boo a note with the fishing pole. When he delivers his tirade about leaving Boo alone, he mentions the "asinine game he had seen us playing" and they were not to "make fun of anybody on this street or in this town." That line shut Jem up right away.
The second thing that Scout mentioned in Chapter 4 was the laughing she heard in the Radley house. Not until the end do we know that Boo had been watching them all of this time and that they provided his daily entertainment. Boo was amused at Scout rolling up to his porch. But she doesn't realize this until the last chapter when she's on his porch.