The person whom the speaker of the poem is addressing is never actually specified. However, the curious invitation that is offered with the unorthodox images would perhaps suggest that the speaker is addressing himself or a part of himself:
Let us go then, you and I,
When the evening is spread out against the sky
Like a patient etherised upon a table...
Perhaps we can see this rather surprising simile to reflect back on Alfred Prufrock and his own sense of oppression, awkwardness and feeling trapped and hemmed in. Does this image represent his own perception of his reality and existence? This would support the idea that the "you" that is addressed is some part of Alfred Prufrock's psyche.
In T.S. Eliot's poem, "The Love Song of J. Alfred Prufrock," I believe there is a good deal of room for interpretation here. I see that two possibilities present themselves: the speaker may be speaking to a woman or to a friend, but I find myself leaning more to one than the other.
If the speaker is addressing a friend, he might suggest:
Let us go and make our visit.
In the room the women come and go
Talking of Michelangelo.
eNotes suggests interpretation that would support (though not exclusively) the idea of two men going out on the town:
The phrase "one-night" refers to hotels where lovers meet in secret, and the reference to "oyster-shells" carries with it the connotation of sexuality, as these are a food said to improve sexual stamina.
"Let us go and make our visit" could refer to traveling to haunts (places that people frequent), to visit with women there.
However, eNotes also acknowledges that "you" is unidentified. And in this case, to theorize as to the identity of the "you" in the poem, it must be read beyond the first stanza. Later in the poem:
And the afternoon, the evening, sleeps so peacefully!
Smoothed by long fingers,
Asleep . . . tired . . . or it malingers,
Stretched on the floor, here beside you and me.
Should I, after tea and cakes and ices,
Have the strength to force the moment to its crisis?
This description, including "you," seems to refer to time spent between the speaker and a woman, stretched out on the floor.
Later still in the poem:
And would it have been worth it, after all,
After the cups, the marmalade, the tea,
Among the porcelain, among some talk of you and me...
Once again, the speaker refers to "you and me." The conversation alludes to "you" again, however in this case, it could refer to their conversation, but also might be describing talk—perhaps gossip—of the speaker and the woman, the "you and me."
Though the poem speaks of going out with "you," it does not have to refer to a man more than a woman. The very lines, "Do I dare?" and "Do I dare?" could refer to the question either of the two might ask of themselves when deciding to go: and for a woman, it would be perhaps exciting and scandalous to go with the speaker, but does not mean she will not go.
Poetry can be perceived to mean different things to different people based upon the personal experiences each person brings to the poem. At the start of the poem, it seems that Eliot could be entreating a friend to go out for a night on the town, however, with so many references to "you and me," and moments they spend together, and with others, I feel inclined to believe that "you" is a romantic interest, not just a friend. (There is much more to this poem, but I have only concentrated on this specific element.)