If he asks his question, he is afraid they won't understand him, or they will essentially shrug him off as having nothing important to say. He worries that he will work himself up to make a grand, prophetic statement, and no one will listen to him anyway, or truly understand what he has been trying to say.
He seems to take comfort in the fact that he is a follower, not a leader. As he says, "No! I am no Prince Hamlet, nor was meant to be." He feels that even though he hasn't ever done anything important/significant with his life, maybe that's okay because he was never an important or significant person anyway. No one ever expects much of the "attendant lord" in the play. So he can "grow old" and "wear the bottoms of [his] trousers rolled," and forget about the dreams he had of being important in the world.
Much like the flame from Dante's Inferno (quoted at the beginning of the poem), Prufrock feels confident in telling his story because he knows that nothing will ever really come of it anyway.