Crooks says this after suggesting to Candy and Lennie in chapter 4 that they should leave his room. At this point, he had just been verbally wounded by Curley's wife when he tried to get her to leave. She wouldn't leave and he said he would tell the boss not to let her in his place anymore. She threatened him, and he then backed down with several "Yes ma'am"s.
When Crooks makes the comment in your question, he was trying to have control over something in his life, and was hoping that at least these two acquaintences would respect his wishes to be alone. After all, the rest of the men were coming home from the bar and Crooks didn't want them to think anything was going on between the three of them for fear that the other men would do something to him.
Personally i feel that Crooks is constantly feeling insecure about himself. This is highlighted upon by the strong theme of racism throughout the story. Therefore he feels that everyone at any disadvantage deserves to have his rights and not be treat any differently. He feels no less inferior than anyone else hence he questions himself why he is treat differently in comparison to anyone else. Within the quotation it states 'even if he don't like 'em', this shows us how needy CRooks is to have his own willpower and right amongst the others. Showing us that he does not care even if the right he had was very insignificant and miniscule in comparison to the greater picture, as long as it is a right of his own.