What's interesting about this conversation is that Jem displays the institutional racism of the town while at the same time showing how he has not internalized what racism really is. He equates himself and Scout to any "colored man" in the quote:
"'You’ve ruined ‘em,” said Jem. 'Why don’t you get a colored man?' There was no note of sacrifice in his voice when he added, 'Or Scout’n’me, we can help you.'"
As an innocent, Jem is not actually racist himself. He and Scout are just as eligible for the job. Still, his underlying assumption that it is not a job for a white man shows how the town's racism has infected him.
Scout echoes this ingrained racism when she criticizes Jem's mudman. In her mind, a snowman is only valuable when it is made of snow—that is, when it is white.
Miss Maudie is outside in the cold when the children go and talk to her. When Jem finds that she has been out in the cold all morning trying to get her garden cleaned up, he takes her cold and dirty appearance in, and suggests that she hire "a colored man" to do the job for her. Jem does not consider that the colored man would be just as miserable out in the cold, nor does he suggests merely hiring "someone" to do it. It is indicative of racism that Jem would suggest that such an undesirable task be relegated to a man of race.