The narrator does note that "many a night, just at midnight" he feels an immense and overpowering terror. This suggests that he has some psychological problem that causes him to feel this way. It is a terror so it is a fear. He notes that it is not a pain or grief; it is simply a fear that arises from the suppressed corners of his subconscious or as he notes, from his soul. Within the context of the paragraph that this admission comes from, it is ambiguous just what the cause of this fear is. But consider that he also talks about the old man's fear in this same paragraph/context. At this point, the old man's fear is that there is someone or something in the room watching him. The old man tries to convince himself that it is the wind or a mouse. Since the narrator talks of his waking fear at midnight in the context of the old man's fear of being seen or watched, we (readers) might suppose that the narrator has a similar fear of being watched.
This would make sense considering that the narrator's real problem with the old man is his eye. So, in the particular context of this story, the narrator could be afraid of being watched by the old man's eye; but in a context greater than the story, perhaps the narrator has a more general but still overwhelming fear of being watched. In either case, this shows how delusional and crazy the narrator is. He tries to justify his madness by illustrating how cunning he is but this is obviously another symptom of madness.
Note that the narrator has no problem with the old man at all. In fact, he claims to love him. And yet he lives in fear of the old man's eye. This is psychotic behavior. After watching the old man in the darkness (thereby projecting his fear onto the old man), the narrator's lantern light falls upon the old man's eye:
I saw it with perfect distinctness--all a dull blue, with a hideous veil over it that chilled the very marrow in my bones; but I could see nothing else of the old man's face or person: for I had directed the ray as if by instinct, precisely upon the damned spot.
The narrator is fixated on the old man's eye. He is mortified at the idea of being watched and yet he literally is focusing on the old man's eye as if, in his madness, he seeks out the thing that makes him afraid. (Also note that at the end of the story, the narrator is plagued with fear at hearing the old man's beating heart. Clearly, this sound is in his mind; again, he seeks out or, in this case, imagines the heartbeat even though it is that which causes his fear. In this respect also, the madness of the narrator causes him to focus on and/or seek out that which makes him afraid.)