I would argue that the narrator is jealous of Robert and the importance that his wife takes in her relationship with him, but that he is actually blind to his jealousy and covers it up with a show of nonchalance that he conveys in his narration. For example, consider the narrator's reaction to the poem that his wife had written about the experience of having her face touched by Robert--a clearly intimate and deeply meaningful experience for her:
When we first started going out together, she showed me the poem. In the poem, she recalled his fingers and the way they had moved around over her face. In this poem, she talked about what she had felt at the time, about what went through her mind when the blind man touched her nose and lips. I can remember I didn't think much of the poem. Of course, I didn't tell her that. Maybe I just don't understand poetry.
What is interesting about this response is that the narrator doesn't like the poem, but then goes on to justify his dislike rather than honestly admitting that he doesn't like this poem because he is jealous of the relationship that his wife has with Robert and its depth of intimacy, which is something that he himself has not experienced with his wife. We can see a similar example of self-deception at the beginning of the story when the narrator tells us that he wasn't "enthusiastic" about Robert's visit, then goes on to say that this is because of the way blind men are portrayed in films. Such quotes show us that the narrator is clearly jealous, but he is not himself aware of the extent of his jealousy.