The reason everything the boy loved, "looking at horses and smelling things, hearing niggers laugh and everything," is because the honesty and purity of it were ruined when he saw Jerry Tillford act the way he did that morning with the "bad women" at the "rummy house." Everything he associated with horses, particularly their honesty and purity was ruined.
The boy saw Jerry as a man that understood the same things he did, that loved the horses in the same way and loved the same qualities in them that he did. This man helped Sunstreak become what he was and in doing so must have understood the power in those horses and the honesty and joy that came in working with them and watching them run.
So when the boy saw him at the "rummy house" and felt the disgust and hatred at the low down meanness and dishonesty that Jerry showed, it took all the joy out of the world of horses and men that the boy thought he knew before.