In the South, the city that is the county seat has a courthouse in the town square. As the center of activity, this square often has older, retired men who gather and sit on the benches there where they discuss "the old days" and gossip about what is currently happening and who is involved. In Harper Lee's novel, the Idler's club is such a group of men who frequent the site of trials, taxes, titles, etc.
On the day of Tom Robinson's trial, Scout gets separated from Jem and finds herself amid this Idler's Club, a group of
...white-shirted, khaki-trousered, susperndered old men who had spent their lives doing nothing and passed their twilight days doing same on pine benches under the live oaks on the square.
Scout overhears them talking about her father, "...thinks he knows what he's doing." Another tells the men, "...the court appointed him to defend this n***." But, another counters that Atticus actually intends to defend Tom: "That's what I don't like about it." Having overheard this, the innocent Scout is rather confused about their comments.