Crooks feels his exclusion particularly keenly. He knows that the only reason why he's excluded from the other men's poker games is because of the color of his skin. It's the same reason why he's not allowed to share a dormitory with them. This is California, not the Deep South, and yet racial segregation is as deeply entrenched here as anywhere else in the country. And Crooks feels its degrading effects every single day of his life.
Candy's also an outsider, separated from the other men by his disability. As far as the others are concerned, he isn't "one of the boys." The men are like schoolboys in that they automatically exclude those who aren't like them. There's also something animal-like about the way they reject those among them displaying the slightest hint of weakness. Perhaps they have an unconscious desire to protect the integrity of the group, feeling somehow that the old, decrepit Candy represents a threat. In any case, Candy's exclusion simply serves to underline the sense of isolation he feels on the ranch and how different he is from everyone else.
As for Lennie, he doesn't really understand his exclusion from the poker games—but it's almost certain he doesn't understand what poker involves anyway, being that he has serious learning difficulties. The other men won't accept Lennie as one of them for the same reason as Candy: he's disabled. Only this time, it's a mental rather than a physical disability that's the main reason for exclusion.