There are contradictions or paradoxes in this story. The sniper has characteristics of innocence as well as violence. He is, or once was, a student. But engaged in the Irish Civil War, he has become a competent soldier. He still has some of that innocence and respect for human life. But he must suppress it because on the field of battle, he must condition himself to put those feelings aside to concentrate on killing the enemy in order to keep himself alive. Thus, he has no immediate remorse when he kills the old woman and the man in the turret. He is in a disciplined mode of self-preservation and carrying out orders. 

When he kills the enemy sniper, he does utter a "cry of joy." He is proud that his ruse has worked in tricking his enemy. But his joy is mostly the result of the preservation of his own life. He is fighting for his life. If he does not kill the enemy sniper, then it is possible that he will be killed. So, the joy is a relief that he has, once again, prolonged his own life by dispatching his enemy. 

Upon seeing his enemy double over and thud on the street below, the sniper's humanity resurfaces. This is when he feels that remorse. The sniper is in a kind of "back and forth" development. Prior to the war, he'd had "the face of a student." This suggests his innocence. Once adapted to the war, he loses that innocence and becomes "used to looking at death." He kills his enemy and experiences joy and relief. But then his humanity comes back and the remorse is there. This repeating contradiction shows the conflicted mental aspect of war, especially a civil war.