Jem shows tremendous insight here and demonstrates his continual maturity. He understands that the group represents danger when his sister and father fail to appreciate this--Scout because she maintains a childish innocence and Atticus because he, perhaps too innocently, believes that the people of his town will ultimately behave appropriately. Just as Atticus brushes aside Bob Ewell's threats later in the novel, he seems to underestimate this threat at his very door.
This is an excellent example of the difference in perception and understanding between Scout and Jem as they grow. Jem, who has begun asserting himself as Scout's elder, is in a position of understanding the adult world and what will take place later in front of the jail. Scout, thankfully as it turns out, lacks this understanding of the ugly side of Maycomb.
Jem is nervous because he thinks they are going to hurt Atticus. These men are upset about Atticus defending Tom, and he's afraid the discussion will turn violent.
Yeah ^^ What they said is pretty much all the same thing, so I hope ya got ya answer :P
He feels that a gang has come to hurt Atticus because he is defending Tom Robinson. Jem worries about Atticus doing this trial. He also knows the townspeople aren't happy about Atticus doing a thorough job of defending Tom.
He is concerned about the welfare of his father, Atticus. A group of men will not necessarily listen to reason for Jem is of an age when he can understand that they would let their anger get the best of them. Atticus has put himself in a vulnerable position by choosing to defend Tom Robinson.