Jem shows tremendous insight here and demonstrates his continual maturity. He understands that the group represents danger when his sister and father fail to appreciate this--Scout because she maintains a childish innocence and Atticus because he, perhaps too innocently, believes that the people of his town will ultimately behave appropriately. Just as Atticus brushes aside Bob Ewell's threats later in the novel, he seems to underestimate this threat at his very door.
This is an excellent example of the difference in perception and understanding between Scout and Jem as they grow. Jem, who has begun asserting himself as Scout's elder, is in a position of understanding the adult world and what will take place later in front of the jail. Scout, thankfully as it turns out, lacks this understanding of the ugly side of Maycomb.