He says that he's invisible because people refuse to see him and that it's due to the construction of their "inner eyes" rather than their physical eyes.
The type of invisibility the Narrator has isn't the kind that a ghost has. Instead, he's invisible because other people don't see or acknowledge him. To them, he's a type, part of the scenery, or just not worthy of consequence. He's a black man living in a time in America where black people were seen as unimportant and unequal to others. That's why, for example, he's forced to fight to win a college scholarship when he goes to give his graduation speech to "the town's big shots."
Ralph Ellison writes:
I am invisible, understand, simply because people refuse to see me. Like the bodiless heads you see sometimes in circus sideshows, it is as though I have been surrounded by mirrors of hard, distorting glass. When they approach me they see only my surroundings, themselves, or figments of their imagination—indeed, everything and anything except me.
Nor is my invisibility exactly a matter of a bio-chemical accident to my epidermis. That invisibility to which I refer occurs because of a peculiar disposition of the eyes of those with whom I come in contact. A matter of the construction of their inner eyes, those eyes with which they look through their physical eyes upon reality.
The narrator has to work to find his own identity and purpose in a society that rejects and exploits him throughout the novel, mainly because of his skin color. As a black man, he's invisible to the majority-white society around him.