This question epitomizes the upbringing and idiosyncrasies of Charlotte growing up. This expression is directed to Zachariah. A man of a lowly status, he is expressing his frustration by telling the story about how he was so cruelly treated by Jaggery.

Now, we have to understand the state of mind of Charlotte at the time she makes this statement. She is duly influenced by the tenets of patriarchy, classism, intolerance, and racial differentiation that she grows up with as a young lady of status. As such, her education, as well as the way she has been brought up understanding life, reflect the views of her father, his friends, her own friends, and all the other set of upper-class acquaintances in her family's immediate circle.

"I don't believe you!" I exclaimed. "Justice is poorly served when you speak ill of your betters." It was a phrase I had heard my father use many times.

Charlotte essentially is saying that Zachariah is lying, which she believes on the basis that he is not a white, rich, titled, young man. If he had those qualities and told his story in society, people (Charlotte included) would have been outraged. Instead, she tells him straight to his face that he should not only refrain from talking about his "betters" but, also, that the story that he is telling holds no value whatsoever simply because it is coming from him.

As such, what the statement also means is that Zachariah will likely also get in trouble if he dares to speak against Jaggery. How dare he? He may be getting mistreated, but he also depends on the mercy of this tyrant to survive. The mentality of the likes of Charlotte would have aligned with this latter, sad statement. Still, this was Charlotte speaking prior to her "conversion" from her previous mindset into reality.