Orwell does not come out and explicitly say "here's the class of Burmese that I despise most of all." But I think the correct answer to your question is probably the Buddhist priests.
Right at the beginning of the essay, he talks about how annoying it was to be in Burma because so many of the "Burmans" hated the English and liked to taunt them if they could get away with it. He talks about how one would trip him and then everyone else would laugh.
But then he says that the "young Buddhist priests were the worst of them all." He says that all they ever did was hang around and make fun of Europeans.
Even though Orwell saw himself as the champion of equality and the mouthpiece for the working class which was often taken advantage of by totalitarian regimes, in "Elephant," his disdain for the Burmese commoners is obvious. He refers to them with derogatory language and even though he does sympathize with Burma in general in regards to their being subdued by the British Empire, he also does not seem to really understand the Burmese commoner's lot in life when it comes to the destruction caused by the elephant and what that means to them. He also seems quite passive about the death of a human (again, a Burmese commoner) in contrast to his mourning the death of the elephant.