Animal Farm is a dystopian text because it portrays a world where the characters seek to have a perfect or utopian society, but their plight results in a world that is worse than the world they changed. 

It is evident from the beginning of the text that the animals' attempt at a new society based on the principles of Old Major would fail because Orwell refers to their endeavors as a revolution. By definition a revolution is one full (360 degree) turn. This implies that the animals' attempts to better their lives will result in facing the same predicaments and problems they faced initially. 

Old Major's ideals about animals working for the betterment of animals rather than the humans, leads to the formation of the idea of Animalism. Unfortunately like most revolutions, the ideals that form the revolution are often distorted to serve the purposes of the individuals leading the revolution. In this case, Napoleon the Pig and his initial ally Snowball, seek for the betterment of the pigs and not all the animals as a whole. Eventually, Napoleon drives Snowball from the farm because he sees Snowball as a threat to his autocratic rule. 

The animals on the farm are easily manipulated because of their belief and blind adherence to the principles of Animalism. What begins as a series of commandments

THE SEVEN COMMANDMENTS

1. Whatever goes upon two legs is an enemy.

2. Whatever goes upon four legs, or has wings, is a friend.

3. No animal shall wear clothes.

4. No animal shall sleep in a bed.

5. No animal shall drink alcohol.

6. No animal shall kill any other animal.

7. All animals are equal. (chapter 2)

and mantras "Two legs bad, four legs good!", evolves into a list of rules that are modified to benefit the pigs yet must be adhered to by all animals on the farm; this leads to the animals' downfalls.

Ultimately, the pigs who are lead by Napoleon betray all of the other animals on the farm. This is evidenced by the ultimate betrayal: the slaughter of the loyal horse Boxer. By the end of the text, the commandments are all modified with the last commandment reading, "All animals are equal, but some animals are more equal than others." The pigs reside in the farmhouse, a symbol of the previous human regime, wear clothes, sleep in beds, and drink alcohol. Ultimately, the animals cannot tell the difference between the pigs and the humans who they trade with, as Clover observes through the farmhouse window, 

"No question now, what had happened to the faces of the pigs. The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which" (Chapter 10, 118).

The animals who could remember life before the revolution know that their life was worse now than before. The other animals did not remember a time without pigs in charge and dogs serving as police and guards. 

Ultimately, the animals in Animal Farm seek a perfect society or a utopian society where there would be no classes, no leaders, and supreme happiness. What they get is a world that is worse than they could have imagined, degraded, and headed to its ruin. 

It might be intereting to start with a comparison of utopia with dystopia.  In the original Greek, utopia means "nowhere"; most uptoias have been written about places that are nowhere --- they don't exist, and it's unlikely that they ever could exist.  An interesting example is the latest Utopian novel that I am aware of, "Walden II" by B. F. Skinner.  This novel presents an idealized community based on Skinner's faith in behavior modification and the application of the scientific method to creating a society (don't just talk about it --- try it and modify if necessary).   Many of my students have read this book and suspect that it might be a dystopian novel in disguise, but you'll have to make up your own mind.

Based on this definition, I take dystopian to be the opposite of utopian --- somewhere (the original Greek meaning is "bad").  They are places where the ideal has been replaced by the shocking reality of what could be ... a "possible" place, usually ruled absolutely.  Take the world of 1984 where Big Brother wields absolute power in the first world that does not pretend to use power for any beneficial end for its citizens ... it uses power not because it does good things, but because it's there and it can.   A similar reality presents itself in Brave New World where the rulers control reproduction and keep its citizens in control without the use of external power ... it uses mindless entertainment, free sex, and drugs and let the people "control" themselves.  These are clearly places that have a "somewhere" about them that utopias don't have.

Animal Farm is a dystopia because it is a "somewhere" --- a place that could be and probably has been, where, in the words of it's sister piece 1984, whoever controls the present controls the past; whoever controls the past controls the future. There is no brute forced used in controling the farm; Just present the truth you want the people to believe and let it go from there.

I would define a dystopian novel as one in which an author creates a world that is a total nightmare to live in.  A utopia is a perfect world, a dystopia is the opposite.  In this novel, Napoleon creates a world that is not exactly the worst possible, but pretty close for the animals.

The world that Napoleon creates is one where the animals (except for pigs) have no power.  They have to do what the pigs say and they are liable to be killed if they do not.  The pigs use the other animals for their own gain.  This is no better than it was when the humans ran the farm.  In fact, it is worse because now the animals have had hope (they though Animalism would improve their lives) only to have that hope completely dashed.

So this is a dystopian novel because Orwell depicts a world that is really unpleasant to live in.