The Idlers' club is a group of who stand idly around the court house, commenting on the action that is occurring. It can be inferred that these men have nothing else to do with their time, so they are retired, close to the ends of their lives and/or cannot work for some reason. Scout overhears these men talking on the day of Tom Robinson's trial. She hears one of them explain that Judge Taylor appointed Atticus to take Tom Robinson's case. Scout has been under the assumption that Atticus had chosen to take the case.
As explained in other answers, the Idlers' Club consists of old men, in their 'twilight years' who like to hang around the town judicially discussing important town business amongst themselves. The court is a favourite hangout of theirs, and at the start of Tom Robinson's trial, which has attracted so much interest, they appear somewhat resentful of the large crowds encroaching upon their treasured space.
The Idlers' Club does not appear in a particularly favourable light. These old men hang around idly, commenting on other people and other people's business. In short, they appear as a group of idle busybodies, town gossips. From the conversation that Scout overhears, in which they discuss Atticus, it seems they have no enlightened opinions to offer; they clearly subscribe unthinkingly to the ingrained racism of the town, for a start:
“Lemme tell you somethin‘ now, Billy,” a third said, “you know the court appointed him to defend this nigger.”
“Yeah, but Atticus aims to defend him. That’s what I don’t like about it.”
Their narrow prejudice is clear from their use of the derogatory term for blacks and also the fact that they clearly don't approve of Atticus doing the job that he's been appointed for: defending the black man Tom Robinson. Such an attitude puzzles the innocent young Scout.
The Idlers' Club is described as:
...a group of white-shirted, khaki-trousered, suspended old men who has spent their lives doing nothing and passed their twilight days doing the same on pine benches under the live oaks on the square.
These men make it their business to hang around outside the courthouse and observe what happens in the court, thus educating themselves about the law so well that Atticus states that they know as much about legal proceedings as the Chief Justice (although whether or not this statement may have been sarcastic remains to be seen). Their voices are detailed as intending to sound "casually important."
Jem and Scout overhear the Idlers' Club discussing Atticus as a large crowd gathers in the first-floor hallway. They argue back and forth, with one suggesting that Atticus only "thinks he knows what he's doing," while another states that Atticus is indeed "a mighty deep reader." They are particularly consumed by the news that Atticus not only has been appointed by the Judge to defend Tom, but that he also wants to do so; this seems to unsettle the group.
Overall, they are an unlikeable bunch who clearly have retained the racist values of the South and who have no qualms about sharing those disgusting opinions and tidbits of gossip with each other.
The Idlers' Club was a group of older men of the town who took it upon themselves as interested in what was going on while pretending to be nonchalant about it. This group is the stereotypical "old boys club" found in every small town. They have seen the town change throughout the years, and tend to resurrect the history of the past when discussing present day happenings.