Well, we need to remember that the narrative choice that Poe made to relate this chilling tale presents us with a profoundly unreliable narrator who himself expresses a series of emotions and views. Let us remember that he stresses the way that he loves the old man and that there was no element of revenge or greed in his desire to kill him. Yet at the same time, you are right in indicating that the way that the narrator kills the old man, producing terror in him and enjoying this fact, would contradict this earlier assertion that he loved him, and it definitely gives rise to a callous tone. Note the following quote when it is clear that the narrator delights in the terror he causes the old man before killing him:
Presently I heard a slight groan, and I knew it was the groan of mortal terror. It was not a groan of pain or of grief--oh, no!--it was the low stifled sound that arises from the bottom of the soul when overcharged with awe... I knew what the old man felt, and pitied him, although I chuckled at heart.
Note again how we are presented with this seeming contradiction as the narrator says that he "pitied" his victim whilst at the same time "chuckling" at the terror he is inducing. It is clear that the manner of killing creates a callous and disdainful tone, as the narrator could be likened to a cat playing with a mouse, deliberately terrifying it, before killing his victim.