The adults could be viewed as the enemy in this book for a few reasons, but I think the adults should be viewed as the enemy for one main reason above all other reasons. The adults are the enemy because they are the puppet masters that are tricking, controlling, and manipulating a bunch of children. To be fair, the Buggers are a viable threat, and Orson Scott Card does an admirable job of selling readers the idea that children are the world's best hope when it comes to commanding the ships being sent to destroy the Bugger home world. That doesn't change the fact that in the case of most of the adults in the book, the end justifies the means. Each child is hardly ever viewed as an actual child. To men like Graff, Anderson, and even Mazer Rackham, the children are basically tools.
Individual human beings are all tools, that the others use to help us all survive.
Fortunately, some of the more astute children realize this fact and actively look for ways to counter the manipulation or even make it work in their favor.
I'm not going to let the bastards run me, Ender. They've got you pegged, too, and they don't plan to treat you kindly. Look what they've done to you so far.
For really good evidence of how the adults are making intentional efforts to manipulate and control Ender and the rest of the kids, look to the starting sequences of almost any chapter. These brief sections give readers a little insight into the conversations that are happening with men like Graff. What makes these conversations a bit more evil is that the men realize their actions are morally suspect, yet they do them anyway. Chapter 4 begins with this exchange:
"With Ender, we have to strike a delicate balance. Isolate him enough that he remains creative—otherwise he'll adopt the system here and we'll lose him." ...
"Sometimes I think you enjoy breaking these little geniuses."
"There is an art to it, and I'm very, very good at it. But enjoy? Well, maybe. When they put back the pieces afterward, and it makes them better."
"You're a monster."
"Thanks. Does this mean I get a raise?"