The men go to the jail because they want to lynch Tom Robinson -- to kill him rather than letting him go on trial. This was something that Southern whites did relatively often to black men who were accused of sexually molesting white women in the early 1900s.
What convinces Cunningham to leave is that Scout talks to him and asks him about the case that Atticus is working on for him. When she does this, she reminds him that he is an individual rather than just part of a mob. He realizes that what they are doing is wrong when he has to think about it rather than just doing what everyone else is doing.
Lynching black men who "supposedly" raped a white woman was a common practice in the south and seldom led to any consequences for those who perpetrated it. Mr. Cunningham and the men with him were in a mob frenzy and had talked themselves into the idea of hanging Tom Robinson. Atticus had been concerned that they might take such an action and with the sheriff unavailable he had decided to monitor Tom's safety.
Mr. Cunningham changed his actions after Scout started talking abut his son being her classmate. Scout brought a sense of reality back into his head and caused him to dissipate the mob of men. In her anger at the men grabbing Jem and handling him harshly Scout had seen Mr. Cunningham and she questioned him if he knew who she was. She could not understand how someone who was supposed to be their friend could behave in such a manner.