T.S. Eliot mentions a few details about the actual people at the tea parties, and then the rest is left up to you, the reader, to infer (or guess).  The only direct mentioning of people there are descriptions of women.  He mentions that "The women come and go, talking of Michaelangelo," and repeats this a couple times.  This signifies that there are groups of women, who are talking about trivial things; kind-of like a group of girls now, talking about the latest band or t.v. show.  The chit-chat is shallow, trivial, and monotonous.  He also describes the women's "arms that are braceleted and white and bare" and an even greater detail about their arms, that "in the lamplight, downed with soft brown hair."  He talks about their perfume, about their trailing dresses, and about how he is intimidated by them, and feels like a bug "pinned and wriggling on the wall" under their scrutinizing eyes.

So, at the party, even though I am sure there is a great mix of men and women of all types, he focuses in on the beautiful-yet shallow-women that he finds so alluring and intimidating; perhaps, the woman he wants to ask his "question" falls under this category.  But he is worried that if he does ask his question, or if he talks about something serious or profund (which he feels would be out of place in the setting of light tea-party talk), he would be so embarrassed and rejected, that he would have to leave.  As he leaves, he worries that they will scrutinize the "bald spot in the middle of my hair" and "how his legs and arms are thin."  He is overly self-conscious about his age, about being seen as a "fool", as unattractive, as unfashionable ("I shall wear the bottoms of my trousers rolled"), and most of all, about being rejected.  He worries that the woman will be irritated, and say, "That is not what I meant at all," in reference to some communication between them.

So, whether these women actually do notice his self-consciousness, his age, his inability to communicate effectively, and his desire to be accepted and profound, this is how he feels they perceive him.  As all people with low self-esteem, Prufrock probably worries about what other people think about him, much more than they actually do think about him.  I hope that these thoughts help; good luck!