The answer to this question can be found in Chapter Fourteen of this excellent coming-of-age novel. In this chapter, Scout overhears a conversation between Atticus and her Aunt where Aunt Alexandra advises Atticus that the time has come to fire Calpurnia. This conversation happens after Aunt Alexandra finds out that Calpurnia is going to take Jem and Scout to her black church. Note what Aunt Alexandra says to try and support her viewpoint:
Atticus, it's all right to be soft-hearted, you're an easy man, but you have a daughter to think of. A daughter who's growing up... You've got to face it sooner or later and it might as well be tonight.
Aunt Alexandra clearly believes that such a close relationship with an African-American is not desirable, especially for a young girl such as Scout who is growing up very quickly. Given the harsh realities of discrimination and racism in Maycomb, we can understand her perspective, even though her views are incredibly unpalatable from our 21st century viewpoint. Social standing and respectability at that time was built on a very clear understanding of the difference between blacks and whites.