According to Jem, a person is "colored" if there is anyone in his parentage or ancestry who is "colored."
Jem, of course, repeats what he has been told or has overheard. Within the setting of To Kill a Mockingbird, which is the 1930's in Alabama, there was an anti-miscegenation law which prohibited interracial relationships. (This law was ruled unconstitutional in 1967 by the U.S. Supreme Court in Loving v. Virginia.)
When Scout asks her brother if they are "colored," he tells her that Uncle Jack has said that as far back as he can trace the Finches they are not,
"...but for all he knows we might have come straight out of Ethiopia in the Old Testament."
Scout concludes, "Well, if we came out durin' the Old Testament it's too long ago to matter." and Jem agrees, "That's what I thought."
This conversation between Jem and Scout has arisen because of the presence of Mr. Dolphus Raymond, who has apparently violated the law with his interracial relationship. Evidently, his offense is ignored because the townspeople believe that he has become a drunkard, perhaps because of a tragic incident in his past. Then, when a small boy holding the hand of a "Negro woman" walks toward Jem and Scout, Jem points out that the boy is one of Mr. Raymond's children. "How can you tell?" asks Dill, "He looks black to me." Scout echoes the question and Jem replies that sometimes a person cannot, but "you just hafta know who they are."
Harper Lee's injection of this small episode right before the trial of Tom Robinson is significant as it suggests prevailing attitudes and conditions.