Ender is only merciless when he has to be, which is to say, when his own life depends on it. The first example comes in Chapter 1, when he fights with the school bully Stilson. From an onlooker's perspective, it doesn't seem like Ender ends the fight when he is ahead, it seems like he ends the fight when Stilson is dead (which he is).
Blood from his nose spattered the ground nearby. "It wouldn't be this bad," Ender said. "It would be worse."
In this moment, Ender does not know that he has killed Stilson. All he knows is that he has been bullied his entire life, and will continue to be bullied as long as he allows it to happen. He knows he needs to get a message across, once and for all, that he is not weak.
This scene repeats itself in the Battle School showers, when Ender must fight Bonzo, another bully, in such a way as to be left alone forever. As he remembers it, "all [he] could see was the way Bonzo looked as Ender kicked upward into his groin. The empty, dead look in his eyes" (chapter 12). This is another moment of mercilessness, in the name of survival. Again, Ender is completely unaware that he has killed Bonzo in this fight. All he knows is that he was afraid for his own life, and he was doing exactly what he needed to do to protect himself.
"Merciless" is an ironic descriptor for Ender, because at his core, it doesn't describe him at all. It could be argued that he is forced into becoming merciless by the people around him. It could be further argued that perhaps that is the author's message. When it comes to the survival of a species, the victor must be merciless.