One of the main themes in "The Swimmer" is the gap between social status and wealth. Ned, the protagonist, was once a person of wealth and held in regard in his affluent community. The story takes place after some unnamed series of misfortunes which ended with Ned as, essentially, the town drunk; he either cannot or refuses to remember his misfortune, still believing himself to be a paragon of the social community. It is also possible that Ned, in his drunkenness, has forgotten all the elapsed time between riding out the storm in the gazebo and finding himself on the side of the road; it is indicated that some amount of time has passed, either literally or figuratively.

As soon as Enid Bunker saw him she began to scream: "Oh, look who's here! What a marvelous surprise! When Lucinda said that you couldn't come I thought I'd die."

[...]

The bartender served him but be served him rudely. His was a world in which the caterer's men kept the social score, and to be rebuffed by a part-time barkeep meant that be had suffered some loss of social esteem.
(Cheever, "The Swimmer," shortstoryclassics.50megs.com)

At the beginning of the story, Ned is welcomed and spoken to as a member of the community. At the end, he is ignored, maligned, and treated like a vagrant -- which he might in fact be. The drop in Ned's social standing shows how the community values wealth above all else; his alcoholism is an excuse, as everyone else in the community drinks easily as much and as often. Instead, they consider him useless in propping up their own social status, as he has none of his own; instead of taking pity on him, they cast him out. Thus, Ned's fall is less a reflection on his own decisions and more on the shallow nature of social class structure among the wealthy.