Prior to arriving at the bazaar, the narrator thinks of the entire event like an epic quest. He has put Mangan's sister upon a pedestal. He conflates spirituality with romanticism in a way that makes him think he's like a knight going after some exotic holy grail for his princess (Mangan's sister). Even the name "Araby" conveyed a sense of adventure because of its associations with the East (Arab):
The syllables of the word Araby were called to me through the silence in which my soul luxuriated and cast an Eastern enchantment over me.
The narrator is completely overcome with the notion of Mangan's sister:
My eyes were often full of tears (I could not tell why) and at times a flood from my heart seemed to pour itself out into my bosom.
The narrator, an adolescent, has built up this quest to Araby in his mind. After hearing the mundane and superficial conversations at the bazaar, the narrator is let down. He gets the sense that these people are not there to promote some exotic scene and experience; they are there simply to make a profit.
The narrator had built his adventure up in his mind. After seeing the reality of the bazaar as simply a superficial market, the narrator is completely let down. And this is when he realizes he's also built up this whole mythology about Mangan's sister and his role as her suitor. That's why he thinks of himself as a "creature driven and derided by vanity"; he'd not only built up Mangan's sister and the bazaar into greater things than they were, he'd done the same with himself.