If Animal Farm had a happy ending, leaving the rest of the novel the same, it simply wouldn't be logical. Snowball will still have been banished and Napoleon would still have taken an initially righteous revolution and turned it into a totalitarian society where the animals suffer just as much as they did under Mr. Jones.
However, if we consider that Napoleon changes his ways, brings Snowball back, and tries to make the lives of the animals better, then this kind of happy ending does not really ruin the book. However, since the novel is an allegory of the Russian Revolution, it would be dishonest. And in this sense, a happy ending would ruin the purpose of the novel (a warning against greed and power) because the allegory would not fully represent the Russian Revolution on which it was based.
Also, since the novel is a warning about the perils of power and fascism, a happy ending would make that lesson less obvious. For instance, let's say that Napoleon went from bad to good; thus, he was corrupt but changed his ways and they all lived happily ever after. On one hand, one could argue that this is more of a fairy tale than an allegory of real historical events and is therefore unrealistic. Thus, it loses its credibility as a lesson about real applications of power. It also suggests the possibility that a happy ending justifies the corrupt regime that led up to it. This is an important point because fascists and despots often use the "ends will justify the means" as a justification for how poorly the society is running. A despot will say that things might be bad now but we'll all be better off in the long run. So, to have a happy ending following all of this corruption is not only dishonest; it also justifies that corruption just as the despot (Napoleon for example) might say that the ends will justify the means.
The other reason a happy ending would ruin or damage the message of the novel is that Napoleon and Squealer systematically changed the revolution into a totalitarian state. They subtly changed the commandments to suit their increase in power and greed and in doing so, they slowly changed the perceptions of the other animals, effectively using propaganda to brainwash those animals/citizens. By the end of the novel, many animals had become so brainwashed that they accepted how corrupt Napoleon's regime had become:
It did not seem strange when Napoleon was seen strolling in the farmhouse garden with a pipe in his mouth -- no, not even when the pigs took Mr. Jones's clothes out of the wardrobes and put them on,
Napoleon himself appearing in a black coat, ratcatcher breeches, and leather leggings, while his favourite sow appeared in the watered silk dress which Mrs. Jones had been used to wear on Sundays.
Even if Napoleon changed for the better, he would have to be charged with crimes against animals (analogously, crimes against humanity). Considering all of the executions and manipulations of power, a happy ending would seem illogical, unrealistic, and too easy, too much like a Hollywood ending.