They share the bottom rung of the social ladder. Candy has just one hand. Crooks is not white and has a crooked back. Lennie is mentally undeveloped.
Each have too few friends, too little power, and no real prospects for a change in his situation. Happiness is as far from these fellows as it can be - or it seems that way.
They all are incomplete or broken workers. Candy is lacking a hand, Crooks' back is broken and Lennie is mentally deficient. They are de facto dependants on the system that allows them to survive. They are all incapable of standing alone.
They are also social outcasts among the men. Lennie cannot communicate or function socially, Candy is elderly to the point of being marginalized and Crooks is a black man in a white man's world.
I have always just accepted that they "belonged" to the hired man's club. They are all in the same boat...doing the work the rich guy's tell them to do.