The impression we get from Nick is that Tom was born into money, was gifted with athletic talent, and never had to give two thoughts to those less fortunate than him. This does not necessarily lead to a racist point of view. However, such a lifestyle is conducive to elitism and Tom is no exception. Tom is an elitist. He thinks he is better than most people.
When Nick is talking with Tom, Daisy, and Jordan in the first chapter, it quickly becomes clear that Tom is self-involved, proud, and, yes, a racist. Daisy uses the word "uncivilized" and Tom strikes at this opportunity to bring up the topic of civilization.
"Civilization's going to pieces," broke out Tom violently. "I've gotten to be a terrible pessimist about things. Have you read 'The Rise of the Colored Empires' by this man Goddard?"
Tom continues, claiming this theory of Goddard's has been scientifically proved. He goes on to say, "This fellow has worked out the whole thing. It's up to us, who are the dominant race, to watch out or these other races will have control of things."
Just as Tom thinks he is from an elite economic class, he believes that his white race is the "dominant" race. He is clearly a racist.