Crooks seems to realize, after all the arguments and threats that take place in his little room, that he is not only a second-class citizen here on the ranch but would be a second-class citizen if he shared a place with George, Lennie and Candy. They would all be whites, and their friends and neighbors would be whites, and he would be the only black. He would almost automatically be subordinated to the lowest ranking member of the four. The whites who intrude into his little room off the barn take over the place just as if he hardly existed. It isn't even his room as long as all of them are in it. He sees the problem when George returns from town. George asks:
"What you doin' in Crooks' room? You hadn't oughta be in here."
Why not? Probably because the other men would disapprove of it, including even the boss. Crooks senses he would also suffer discrimination and ostracism because he is black. When Candy is leaving his room, Crooks calls him back and asks:
"'Member what I said about hoein' and doin' odd jobs?"
"Yeah," said Candy. "I remember."
"Well, jus' forget it," said Crooks. I didn' mean it. Jus' foolin'. I wouldn' want to go no place like that."
Crooks seems to realize that he is better off alone. At least his has his little sanctuary for awhile. If he went with George, Lennie and Candy to a different part of the state, he might be the only black man in the whole area. He could be subject to insults from the local inhabitants and feel out of place whenever there were visitors. He is a proud man. Earlier he had told Candy:
"Maybe you guys better go. I ain't sure I want you in here no more. A colored man got to have some rights even if he don't like 'em."
Apparently the "rights" he is referring to include the right to privacy and the right to a little place in the world where he isn't subject to insults and abuse.