There are two ways to answer this question. One way is to look at what Rassendyll himself says about this. The other is to look at it from an outside perspective -- to look at why we think he did it.
Rassendyll says that he is not completely sure why he didn't shoot. He thinks it is partly because it does not seem fair. He doesn't want to be "one of a crowd against him." The second reason he gives (and he says this reason is more important) is that he was curious to see how the scene would play out if he did not get involved.
From an outside perspective, we can infer that Rassendyll does not shoot Rupert because he sees something of himself in Rupert. Rupert is something of a foil for Rassendyll -- he is Rassendyll's bad side. Rassendyll does not want to kill Rupert because he realizes this. He realizes that he is really no better than Rupert and therefore does not feel he has the right to kill him.