There are several reasons why Neddy Merrill might only speak with women. First, he often meets women by the pools while their husbands are inside the house. In addition, he looks to women to provide the antidote to the spiritual emptiness he is feeling. He says of Shirley Adams, a woman he has had an affair with, "Love—sexual roughhouse in fact—was the supreme elixir, the pain killer, the brightly colored pill that would put the spring back into his step, the joy of life in his heart." He relies on physical contact with women as an antidote to the spiritual pain he is feeling, and he clearly has very little real emotional connection with others. He does not even realize that many of his apparent friends have moved, and it is obvious that he lives mostly in his own mind.

Depending on how one reads the story "The Swimmer," there could be several reasons why in his cross-suburbia swim Neddy speaks only with women, never with men. If one reads it with the ending in mind and assumes that the swimmer is really Neddy in the days when he has lost his home, his wife and children, and his social and financial status, then perhaps he only speaks with women because women historically have been the ones to distribute charity in the community toward those who are down and out. Even though at least two of the women he speaks to are unkind to him, presumably their male counterparts would have had even less patience with the former neighbor and friend who has become an alcoholic and beggar. 

If, on the other hand, one reads the story as Neddy's dream, Neddy may sense subconsciously that women will be more supportive and cooperative, or that women will be more likely to give his ego the strokes he is looking for. Women are less threatening to him than men, and so he populates his dream with women in the speaking roles while men are the ones who mock him (the drivers on route 424), expel him from the community (the lifeguards), and rebuff him at the bar (the bartender). 

Finally, if one reads the story as allegory or metaphor, women represent the two things that motivate Neddy: social standing and sexual conquest. As the ones who tend to be the organizers of social gatherings (Lucinda, for example, keeps the Christmas list and Grace Bizwanger is the one who invites her various vendors to her parties), women are the best symbols for social status. Shirley Adams, whom Neddy feels confident will "surely" always be open to his advances, is the symbol of sexual conquest. The men in the story represent the more mundane aspects of life, not the parts that Neddy really cares about. In fact, the men, when they play much of a role at all, represent things that impede Neddy in his pursuits. 

Depending on one's interpretation of the story, Neddy may speak only to women because they are the ones who would be most charitable to a man who has fallen on hard times; because they are less threatening to Neddy's ego; or because they symbolize social standing and sexual conquest, Neddy's two driving motivations.