One of the most significant pieces of evidence that it is the old man's vulture eye that seems to compel the narrator to kill him is that the narrator admits to having no other motive for murder.  He says,

Object there was none. Passion there was none.  I loved the old man.  He had never wronged me.  He had never given me insult.  For his gold I had no desire.  I think it was his eye! yes, it was this!

In other words, he does not look to gain anything by committing the murder: he is not planning to rob the old man of his fortune or to inherit it upon the old man's death.  Likewise, the narrator does not feel any hatred or ill-will toward the old man; in fact, he says that the narrator has never done anything wrong to him or injured him in any way. There is simply no other reason to kill him.

Once he lands on the idea that it is the old man's eye that spurs him to murder, he describes it in great detail, saying, in part, "Whenever it fell upon me, my blood ran cold; and so by degrees -- very gradually -- I made up my mind to take the life of the old man, and thus rid myself of the eye forever."  He believes that it is the old man's "vulture eye," cloudy, likely, as a result of cataracts (a common optical ailment among the elderly) that prompts him to murder.