The "you" in this poem is ambiguous. It could be another person Prufrock is speaking to with whom he is going to the party. He could be talking to himself. He could even be inviting the reader to accompany him on his journey. Whatever the case, T. S. Eliot establishes with this opening line the idea that Prufrock is addressing or talking to someone who never answers back. This places the poem within the tradition of the dramatic monologue: a poem where we as readers "overhear" the narrator speaking to someone else whose response is not recorded. Usually, in this kind of poem, our main focus is on learning about the character of the speaker.
Underlying the ambiguity of who he is speaking to is the fact that Prufrock seems at times to be addressing another person, as in these lines:
To lead you to an overwhelming question
. . .
Oh, do not ask, “What is it?”
Let us go and make our visit.
And yet at other times, his "talk" seems to be an interior monologue
; it's as if we are hearing his thoughts, such as in the repeated:
In the room the women come and go
Talking of Michelangelo.
This is not what a person would normally say to another person, especially not more than once.
I tend to think that Prufrock is going to the party with someone, but he is also often lost in his own thoughts. At times—many times—it feels as if he is talking to himself and forgetting his companion is there.