There was plenty of racism on each side.
On the imperialist side, one major justification for imperialism was the idea of the white man's burden. People felt that Americans were completely superior to the Filipinos and other people they were conquering. They felt that they deserved to have an empire because they were superior. This is clearly a racist attitude.
On the anti-imperialist side, many people did not want to take places like the Philippines because they did not like the fact that the Filipinos were not white. They worried that the Filipinos would then become part of the United States. Taking that many non-whites into the country would, they thought, weaken the United States. This, too, is clearly racist.
I would have to say that I believe that racism was a tool for colonial powers to justify the acquisition of colonies more than it was actually a motivation. In other words, the taking of lands from other people had more to do with the acquisition of resources and the establishment of markets than any aspect of race politics. Having said that, those on the imperial side used race as a justification to the public for expanding empires. The dialogue went something like this: these non-white populations are backward and uncivilized. We as a superior people are burdened with the duty of civilizing these savages with Christianity and Western ideals. The notion of whites making nonwhite people better through assimilation is clearly an example of racism.
At the turn of the Twentieth Century, the debate in the United States was fierce about America's role as an imperialist. There was widespread sentiment that the United States should not force its way of life on other people. There was also a Marxist element that existed in the anti-imperial camp, but many groups felt that American colonization violated the principles by which the republic was founded on. Most felt it was an abandonment of the American ideas of self-government and isolation. There was, however, those in the anti-imperialist side that were concerned that by acquiring new lands, nonwhites would become citizens automatically. They believed that these uncivilized people would migrate to the United States and changed the racial demographics of the nation. This, obviously, is a nativist sentiment driven by racism.