To answer the question accurately, one first needs to define the two systems and then establish the nature of Orwell's criticism.
Socialism is basically defined as "a political and economic theory of social organization which advocates that the means of production, distribution, and exchange should be owned or regulated by the community as a whole."
In terms of Old Major's instructions during the meeting in the big barn, this is the type of society he had in mind for all the animals. Once they were rid of Man, they would be able to organize their lives more efficiently and work together as a group. In this way, every animal would labor equally, and all animals would also equally benefit. This essential aspect is later contained in one of the commandments: "All animals are equal."
After the Rebellion, it soon becomes apparent that this basic tenet is not entirely adhered to. Firstly, the pigs take leadership of the farm and absolve themselves from doing any physical labor, since they are, supposedly, the "brain workers." They undertake the supervision of the other animals and do the administration. This immediately places them in a privileged position, and they take advantage of this.
The first indication of their misuse is when they claim the windfall apples and milk for their exclusive use, stating that they need the nutrition it provides for the kind of work they do. The animals are easily manipulated into believing that if they are not allowed such a privilege, "Jones would come back." The animals, obviously fearing Jones's return, are acquiescent and the pigs, knowing the other animals lack the intelligence to question their motives, start exploiting them for their own benefit.
They systematically change the commandments to suit their purpose and claim more and more privileges. Napoleon secretly trains nine puppies, which he later uses to drive his greatest opposition, Snowball, from the farm and to tyrannize the general animal populace. He becomes Leader, and all the animals have to bow to his authority or face retribution. Napoleon becomes a dictator, and the other animals on the farm are literally his slaves.
Through a system of propaganda, manipulation, and threats, the pigs assume a very powerful position on the farm. In the end, it is obvious that the general animal populace is worse off than in Jones's time. Their dream of a utopian society lies in tatters, for the pigs had shown their selfish and greedy nature and are the only ones who benefit. This aspect is most pertinently symbolized in the single commandment which has replaced all others: "All animals are equal, but some animals are more equal than others." The paradox is clear, but is obviously understood to mean the pigs are better than the other animals.
Capitalism is described as "an economic and political system in which a country's trade and industry are controlled by private owners for profit, rather than by the state."
Orwell criticizes capitalism by first focusing on Mr. Jones's greed. He uses the animals for his own profit. The animals are mere tools in his hands, as expounded by Old Major during the first meeting in the big barn. Man seeks only to benefit himself and treat animals with the greatest ignominy for his own greed and profit. The animals do not benefit from their own labor at all.
After the Rebellion, the animals are not supposed to adopt human vices and become as selfish and greedy as he. This ideal, however, is soon shattered, for the pigs adopt a supercilious attitude and manipulate the animals so they alone may benefit. Whatever the pigs do is exclusively for their own pleasure and privilege. None of the other animals benefit as much as they from the hard work put in by the general animal force.
The pigs clearly have capitalist tendencies and abuse the other animals more than even Mr. Jones did. There is no equal distribution of labor, privileges, or any other benefits. The pigs, who do not perform any physical labor, claim the best of the best for themselves and grow fat. They live in luxury and adopt human characteristics. The other animals become mere servants to cater to the pigs' every need.
The epitome of the pigs' greed is best conveyed by Boxer's death. The pigs have become so ruthlessly greedy that his body is sold to the knacker. The proceeds of this sale are used to buy a case of whisky from which the pigs later get drunk, supposedly while holding a banquet in Boxer's honor. The other animals, obviously, are excluded.
Orwell, in his criticism of both socialism and capitalism, makes it obvious that these systems fail not because they are inherently flawed, but because of Man's innate selfishness and greed. It is this fact which eventually ensures that those in authority become corrupt and exploit, threaten, and manipulate those they deem inferior for their own profit and privilege.
Orwell's critique of socialism is the most obvious in the story, with the animals attempting a communist-style society in the farm and slowly failing as Napoleon takes over as a dictator. The animals initially work with each other, everyone taking a roughly equal share in the labor, but the pigs, being more intelligent, slowly start to move all the actual labor onto the other animals while placing themselves into positions of honorary, and then explicit, dictatorship. The farm becomes a slave-society, with the pigs acting human while the other animals work hard for little reward.
Orwell's criticism of capitalism is more subtle, but it is clear throughout the story that the humans -- who all practice general capitalist policies -- are cruel and abusive. The animals try to escape this through their own work, but the animal society proves untenable and Napoleon begins commerce with the other farms. The farmers first try to cheat the animals, and then start working with the pigs. One farmer jokes:
"If you have your lower animals to contend with," he said, "we have our lower classes!" This bon mot set the table in a roar...
(Orwell, Animal Farm, msxnet.org)
This shows how uncaring the humans (capitalists) actually are, although the animals are certainly worse off under Napoleon than they were under Jones.