Crooks is set apart from the other men because he is black. This story is set during the Great Depression. At this time, racism was still quite common and quite public. Although clearly unethical from our perspective in 2015, it would have been quite common for businesses to segregate black and white workers. In fact, "Separate but Equal" was a legal doctrine (1896) that justified and legalized racial segregation. So, there was nothing Crooks could have done about this. He reluctantly had to accept this was the way things were done.
This is why Crooks is one of the sympathetic characters in the story. In Chapter 4, Crooks is hesitant to allow Lennie and Candy into his bunk. His reasoning is that if they (the white workers) won't allow him into their bunkhouse, why should he be accommodating to them. But he does and they have a civil conversation. However, Curley's wife comes in and Crooks senses that she could be trouble. When he insists that she leave, she tells him to keep his place and that she could have him hanged. Unfortunately, she is correct. The boss and certainly Curley himself would take her side no matter what Crooks said. He knows that because he is black, he is looked upon as a second class citizen by whites. When Curley's wife tells him off, he backs down immediately.
Crooks had reduced himself to nothing. There was no personality, no ego--nothing to arouse either like or dislike. He said, "Yes, ma'am," and his voice was toneless.
Crooks is segregated because at this time segregation was legally and culturally permitted. He accepts his ostracism because he knows this is part of the institutionalized racism that was quite acceptable in America at this time.