There had been a sordid story circulating around about Mr Raymond - his fiancée discovered he had been having an affair with a local black woman. So she went home, put a rifle to the floor, pulled the trigger with her toe, and blew her brains out. This put the icing on the cake, so to speak, since Mr Raymond already had a bad reputation as a "nigger-lover" and drunkard.
Later on the children learn that Mr Raymond was faking being a drunk to spite the community by "confirming" one of the rumours about him. They also discover that Mr Raymond is a kind man who just prefers the company of Negroes, whom he considers to be more "real" and less pretentious than the other people in the white community.
So as it turns out, Mr Raymond has some rare qualities as a person and is not as "bad" as all that.