Orwell provides no explanation for why the pigs and the hens confess to being traitors. At this point, Napoleon is busy discrediting Snowball as far as he can. Napoleon insists that Snowball was secretly in league with Farmer Jones from the beginning. Squealer insists that that Snowball tried to engineer the animals' defeat at the Battle of the Cowshed. Boxer denies this, reminding everyone that Snowball was wounded defending the cowshed. However, when Squealer invokes the authority of Napoleon, Boxer backs down. This appears to signal that a new version of the truth has been sanctioned.
Knowing what we do of the animals, we can try to piece together their logic in confessing. The pigs are intelligent, so it is possible that they confess in a failed attempt to get clemency. They have already been identified as traitors and already attacked by the dogs, only to gain a temporary reprieve due to Boxer's intervention. However, the confession fails to work and the dogs rip the four pigs' throats out.
The hens, we are told, are not very bright. In fact, they are part of the group of animals that can't even manage to remember the Seven Commandments. It is also true that hens, for a time, fought back against Napoleon's plan to sell most of their eggs. It is very possible that the three hens who step forward and confess that Snowball came to them in a dream and told them to rebel believe what they are saying is true. This would be consistent with Orwell's idea that less intelligent people are more likely to be swayed by suggestion into believing ludicrous propaganda. We know that the hens tend to be honest, because they have worked so hard in the past toiling to gather every last wisp of hay and stealing none. They are part of the true believers in the revolution, which it makes it all the more poignant that they find themselves brutally sacrificed to Napoleon's ambitions.
The pigs and hens that are called forth for having challenged Napoleon's leadership mistakenly believe that their confessions will provide them with some mercy. It is the theory confession shows a sense of responsibility. Thus, the leader will have respect for the criminal and therefore grant some leniency. It is a policy of parole boards here in the U.S.. A convicted felon who refuses to admit his crime will not be granted parole.
Unfortunately, Napoleon is a ruthless dictator. He executes all the pigs and hens, despite their confession. You might expect that the fear this would create would keep everyone else quiet. However, paranoia often makes us behave irrationally. The animals become so scared that they will be executed that they begin to confess everything in the hopes of appearing honest and loyal to their leader.