Most of the adults in To Kill a Mockingbird have settled in to their lives, with years of opinionated thought engrained in them. Adults are less likely to change, and many of their beliefs and views about other people around them will stay with them until they die. Racist attitudes and views about various families (such as Alexandra's Streaks and "dicta" about people, see Chapter 13) never go away. According to Scout,
... the present generation of people who had lived side by side for years and years, were utterly predictable to one another: they took for granted attitudes, character shadings, even gestures, as having been repeated in each generation and refined by time.
Jem, Scout and Dill had no such burden. They saw things from an innocence that only children can, without past opinions or knowledge affecting their judgement. They hear gossip and listen to other people's speculation, but they look at things more clearly than most adults. Although they at first fall for the terrible rumors about Boo, they are able to see past them once they receive his friendly gifts. They finally realize that the stories about Boo are just rumors, and that the adults of Maycomb have misunderstood him in a disgraceful way.
The children, like Atticus, are mostly color blind when it comes to race, and the children feel at home at the Negro church and sitting in the balcony of the courtroom. They have no fear of Dolphus Raymond, and they understand him (unlike the other people in Maycomb) once they spend a few minutes getting to know him.
Since Scout and Jem are so young, they don't listen to the gossip that the adults do, so they don't judge people by how they look or what people say, they judge them more fairly.