This isn't an either/or question. Yes, you could say that Conrad's work is racist, but if you dismissed it for that reason, you'd be dismissing it too fast, because yes, there is much more to it.
Conrad's work is racist in how it portrays Africans, and in how it treats Africa as a site of transforming mystery. The darkness of the jungle fuses with the darkness of native skins and the darkness of the human heart. Conrad uses Africans not as characters most of the time, but as symbols of dark passion and savagery.
However, the book is more than simply racist, because he also critiques the system that produces this division, and shows what happens when men live with such attitudes. He also produces deeply poetic visions of madness/passion that remain useful even when we adjust for the racism.