This is a particularly apt question due to the epigraph from Dante’s The Divine Comedy that precedes the poem. In the epigraph, a damned soul from Hell hesitatingly agrees to tell his story, because he believes that no one will ever hear it. In this way, Prufrock can be understood to be trapped inside his own piece of Hell. So, then: what torments him?
The answer is quite simple: himself. Prufrock appears to be his own worst enemy in this poem. He despairs of his alienation without taking steps to alleviate it; he is indecisive:
In a minute there is time
For decisions and revisions which a minute will
He is consumed by a desire to share his life with a woman, but he dislikes the shallowness of the women he is surrounded by, who “come and go” and yet speak only of Michelangelo. In any case, Prufrock believes that he would be rejected should he propose to a woman:
Would it have been worth while
If one, settling a pillow or throwing off a shawl,
And turning toward the window, should say:
"That is not it at all,
That is not what I meant, at all."
Wandering the narrow streets of the city, Prufrock finds a kinship with the “lonely men in shirt-sleeves” who lean out of windows as they smoke pipes. His fondness for wandering through the worse parts of the city at odd hours speaks to his isolation, but Prufrock does not attempt to solve it—he does not speak to anyone outside of his own mind, and imagines spending his old age alone, with only his imagination as company. In this passage, romance personified as mermaids singing in the distance, who sing “each to each” but not to him. In believing that any attempt to connect to people will be fruitless, Prufrock seals his own fate.