The term "colored man" appears several times in Harper Lee's novel To Kill a Mockingbird. While the term is not as prevalent today as it was during Lee's time, the term was one which was readily accepted, and used, during the time the novel was written.
To Kill a Mockingbird was written in 1960. During this period, racism was rampant in America, and predominately in the South. Given that the novel was set in Alabama, it is of no surprise that the novel depicted terms which were used in racist ways (in referring to African Americans-or blacks).
The term colored, therefore, refers to a person of color (meaning a person with black skin). (Therefore, the term colored man refers to a black man.) Whites were the ones who were in power, the ones who decided who could do what, and the ones who regarded blacks as second-class citizens. Even though slavery was abolished in 1865 (with the passing of the 13th Amendment), many whites failed to allow blacks the same unalienable rights which were bestowed in The Declaration of Independence.
Is was not until the Equal Rights Amendment was passed that racist ideology began to dissipate. (That being said, there are still some who believe that African Americans should not have equal rights today.)