As other educators have noted, when he makes this remark, Crooks is referring to the deep racial prejudices that existed in this time and place. Just wanting to engage with other children, neither Crooks nor the white children with whom he played were actively aware of the racial prejudices that existed in their parents’ world. They were innocent and free of such biases.
Nevertheless, there were subtle messages conveyed to the children by their parents. We can see this in the statement that Crooks makes. His father communicated something negative about his playing with the white children, even if Crooks did not realize at the time why his father was so disapproving of it.
As an adult, Crooks certainly understands the racial divide between him and the other men on the ranch. When Lennie comes into his room, Crooks tells Lennie,
I ain’t wanted in the bunk house, and you ain’t wanted in my room.
However, it is far more than just the father’s concern about Crooks encountering active prejudice as an adult that makes him worry about the young child Crooks interacting with white children. After all, whether the father permitted Crooks to play with the white children or not, the racial prejudice would impact his life as he got older. It was also probable that Crooks’s father feared that any small altercation Crooks might have with another child could result in punishment that would be completely out of proportion. It would seem clear that if Crooks got into any dispute with another child, as young children often do, it would be the white child who would be believed. The adult Crooks understands this. He tells Lennie,
If I say something, why it's just a nigger
Given this, in any possible dispute with his white playmates, the young Crooks probably would have borne the punishment, regardless of his actions and even if he were innocent in the matter.