The circumstances of the fight in which Romeo kills Tybalt sway the Prince to decree banishment for Romeo, not death. The prince's thinking becomes clear at the end of act 3, scene 1.

Prince Escalus comes on the scene and gets an explanation for what happened. At the end, Lady Capulet says that because Romeo killed Tybalt, he should receive the death penalty. However, Lord Montague says that in killing Tybalt, Romeo simply did what the law would have done: Tybalt would have gotten the death penalty anyway for killing Mercutio. Romeo's killing Tybalt made up for Tybalt killing Mercutio. Now, everything is even, and Romeo does not deserve to die.

The Prince notes that since Mercutio is his relative, he has a special interest in this case, implying that he approves of Tybalt paying with his life for killing Mercutio. The prince sides with Lord Montague's idea that since everything is even—a life has been exchanged for a life—he will not add to the body count by demanding Romeo's execution. However, he also makes clear that he wants to impose a serious penalty on Romeo for disobeying the new law about street fighting. Therefore, he is banishing him from Verona.

The prince is clearly angry that his own cousin was killed by Tybalt but is satisfied that Tybalt has paid for the crime. The prince shows his wisdom in not applying the death penalty hastily, but thinking through what makes sense and showing mercy, while at the same time also applying a stiff penalty on Romeo.

We are never told explicitly why the Prince does not follow through on his threat to have anyone guilty of fighting executed. However, we can speculate.

First off, the Prince may have allowed Romeo to be merely banished for killing Tybalt because Tybalt had just killed Mercutio. In this sense, Romeo was acting as the executioner himself. Mercutio is a relative of the Prince, and the Prince may have had some sympathy for Romeo and a grudge against Tybalt in this case.

One could even make the case that Romeo kills Tybalt in self-defense. Tybalt was itching for a fight that Romeo did his best to avoid. In fact, Tybalt would have attacked Romeo straight away if Mercutio had not intervened.

Another possible factor may be that the Prince is afraid of further violence should Romeo be executed. Romeo has a reputation for being an upstanding citizen of Verona. Having him killed might just anger the citizens of Verona and turn them against him. He already seems to have a tenuous grasp on his city. Maybe he is afraid that executing the son of one of his city's leading citizens could result in rebellion. At the very least, it could lead to further feuding between the Capulets and the Montagues, something he is trying desperately to avoid.

Overall, the Prince does not come off as a strong leader in Romeo and Juliet. From the very opening scene of the play, he struggles to keep the peace in Verona. This may be because he seldom follows through on his threats and promises. If this is the case, banishing rather than executing Romeo may just be within the keeping of his character.

In Romeo and Juliet, another possible reason for the Prince's reluctance to follow through on the sentence of death that he has proclaimed for anyone who disturbs the peace of the streets of Verona is that he may reason that his putting of Romeo to death may ignite more feuding and simply perpetuate the hatred between the Montague and Capulet family. For, since he knows that already three civil brawls have erupted from only "an airy word" and caused Verona's senior citizens to take out their old weapons and resume their fighting, the Prince may feel that banishment for Romeo is preferable.

In addition, being the Prince, he may have been alerted that Friar Laurence has married Romeo and Juliet; thinking that their marriage may not yet be consummated, the Prince may want to be able to force the annulment of this marriage by keeping Romeo away long enough for such an annulment.  In this way, too, he can avoid more feuding.

Or, he may have only been advised that Romeo was seen at the Capulet home on the night of the masked ball for Juliet.  Then, the Prince may conjecture that if he banishes Romeo, Juliet's eyes will fall upon someone else, such as Paris--who is, after all, his kinsman--and she will not become interested in a Montague, the enemy of her family.

The Prince never offers us an explicit reason for only banishing Romeo and not having him put to death. However, we can infer that perhaps the Prince believed Romeo was justified in taking vengeance on Tybalt for the death of Mercutio. Mercutio was also the Prince's cousin.

Additionally, we also learn from Lord Capulet (of all people)  that Romeo is known around Verona to be a fairly upstanding person with no record of violence in his past. It would make sense then, perhaps, for the Prince to spare the life of Romeo and only banish him.

We are not really told why, so this is one that we have to speculate on.

In my opinion, there are a couple of possible reasons why the prince has been merciful.

The first reason is that he might have heard about Romeo's real part in the fight.  He might have heard Romeo tried to break it up and only got made when Tybalt killed Mercutio unfairly and unnecessarily.

Second, I imagine that the prince does not want to make the Montagues angry if he can help it.  They are powerful, and it seems to me that he might want to try to keep them happy if at all possible.