Consider the life of a segregated African-American in 1930s California. And Crooks is not only segregated from the other ranch hands, but he is the only black man ostensibly anywhere nearby. I think Crooks is both lonely and bored.
He is also intrigued by the fact that Lennie doesn't seem to understand why Crooks is treated differently. He just strikes up a conversation (albeit a naive, innocent one) with Crooks as though the two were not a black man and a white man, but just two men. This is quite likely the only time Crooks has ever felt this way in his life.
Even though you could argue that the men on the ranch were not intentionally cruel towards Crooks, he probably felt the sting of segregation more than most, as he was facing it entirely alone.