these men are ready to lynch Tom. As Atticus prepares to face the angry group,Jem, Scout and Dill run up and join him.

It is easy to understand why Atticus is so frightened that the children are there: he fears that they may be harmed, and he tells Jem to take the other two kids and go home.

…by the way he stood Jem was not thinking of budging.

"Go home, I said."

Jem shook his head. As Atticus's fists went to his hips, so did Jem's, and as they faced each other I could see little resemblance between them…Mutual defiance made them alike.

It doesn't matter if Atticus demands or pleads, Jem will not leave his father's side. It seems to me that more than anything, Jem wants to protect his father, and he takes a stand. Atticus is afraid, but Jem is too young to know to be afraid. However, Jem is of an age to grasp the importance of supporting his father, as his father tries to do the right thing in protecting Tom. I don't even know that Jem understands the implications of the mob's demand, but he can certainly comprehend the angry demeanor of the men who are confronting his father.

When Scout has had a chance to speak to Mr. Cunningham, helping him to remember that like Atticus, he is a father, Mr. Cunningham disperses the crowd. As they start to walk home, Atticus and Jem walk ahead, and Scout is sure her father is yelling at Jem for his disobedience. She realizes she is wrong when he ruffles Jem's hair… 
"his one gesture of affection."

I believe that Atticus, while frightened, is more than a little proud of his son for standing by to support him. Perhaps Atticus is realizing that Jem is becoming a young man, putting his childhood behind him. It may simply be that Atticus realizes as they go back to the house, that they are two men walking—united in purpose—rather than a man and a boy.

I'm going to guess that you are talking about the scene at the jail when the Finch family confronts the lynch mob in Chapter 15. Atticus is alone at the jail, standing guard over Tom, who is inside. Atticus has already heard that there may be an attempt to take Tom forcibly and, sure enough, a number of vehicles arrive with a group of men from Old Sarum ready to take Tom for a necktie party. Things look bad for Atticus, and even worse for Tom, when Jem, Scout and Dill suddenly appear. Atticus orders them to leave, but Jem shakes his head. When a man "yanked Jem nearly off his feet," Scout

kicked the man swiftly... I intended to kick his shin, but aimed too high.

Jem repeatedly told Atticus, "I ain't going," but it was Scout who eventually shamed the men into leaving. Atticus was proud of Jem for his courageous stand, which probably saved Tom's life and Atticus from injury or death. Atticus had always taught his children to think for themselves, hoping they would make the right decision, and this time it worked out for everyone involved.

As they passed under a streetlight Atticus reached out and massaged Jem's hair, his one gesture of affection.