In Ralph Ellison's Invisible Man, the narrator is an unnamed black man telling the story of his youth. He lives in a small Southern town, graduates from high school and is awarded a scholarship to a black college (a scholarship he must claim by taking part in a bizarre battle competition for the entertainment of his town's white aristocrats). When he attends college, he is subsequently expelled for showing a white trustee the reality of black life outside campus.
The narrator then travels to New York, where he stumbles upon the eviction of an elderly black couple and inspires the crowd to rebel against the police carrying out the proceeding. When riots break out in Harlem, he becomes entangled with a gang of looters, who call for his lynching and seal him in an underground coal bin, the location from which he shares his story.
Clearly, the narrator has experienced extensive and often violent racism throughout his life. You likely noticed while reading the story that most of these happenings are caused by white aristocrats, which is what makes the narrator's fear of the sleeping farmer particularly telling. The farmer was not a part of the elite, but he was white. The narrator is fully aware that it did not matter if the farmer was rich, of average wealth, or poor; he was white and therefore had power over the narrator. The racism the narrator experiences goes much deeper than class or wealth, and he is very aware and frightened of it.
He had power over him, although the white man was poor he still had control over him. No matter how poor you was if you were white you still had power over blacks. (ch. 2)