The narrator is unreliable because the story he tells contradicts his purpose in telling it. That is, from the beginning of the story, the narrator wants to prove that he is not insane and tries to do this by explaining how and why he came to kill the old man. But those actions actually show that he is insane. The narrator's insistence that his mind, far from being sick, has actually become more powerful, especially through a heightened sense of hearing, suggests that he is prone to delusions. His obsession with the "vulture eye" of the old man is another delusion brought on by his so-called heightened perception. His desire to kill comes not from anything the old man did but as a way for the narrator to find release from the torment of his senses. Of course, killing the old man does not solve the problem; the narrator continues to hear the man's heart beating even after he has been chopped up and buried under the floorboards.
On the other hand, it's possible to consider the narrator "reliable" in the sense that he is giving a true account of what he thinks he's done. In this case, the story challenges the reader to understand the narrator's actions from his point of view—the point of view of a madman.