The name is sort of a joke, since the word 'idle' suggests wasting time, doing nothing worthwhile. The so-called 'Idlers' Club' of Maycomb is a group of older, mostly retired men who like to get together, talk politics, and just "chew the fat." They often sit in on court hearings, commenting on, even criticizing, its procedure. The interpretation of the law as exemplified by current cases and the way they are handled is their favourite past time.
The Idlers' Club could be considered the masculine counterpart to the church ladies' Missionary Circle. Each has its official objective, but both are an excuse to just get together with members of the same sex to gossip, socialize a bit, and exchange ideas.