The woman in the ermine toque is Miss Brill's double. Once upon a time, this woman was young—though that time has passed, as we learn that her hair is no longer "yellow." Once upon a time, her ermine toque (a small hat made of a very expensive fur) was likely fashionable and thought to be lovely; however, now it is "shabby," just like everything else about her. Even her white glove, which ought to be pristine, looks like a "tiny yellowish paw." Likewise, Miss Brill doesn't realize that she, too, is unwanted by other park-goers and that the fox fur she thinks of as a "little rogue" is shabby and worn as well.
When the woman in the ermine toque bumps into a man with whom she is apparently acquainted, the man is incredibly rude to her: he refuses even to speak to her and blows smoke in her face, flicking away his match while she is still attempting to converse with him. This is similar to the interaction Miss Brill with the young couple who shares her bench. However, when the man leaves her, the woman in the ermine toque "turned, raised her hand as though she'd seen some one else, much nicer, just over there, and pattered away." In other words, either the woman does not understand the rejection she's received and so she moves on or she does feel the sting and pretends not to so as not to have to deal with the embarrassment. Likewise, Miss Brill does not, at first, realize that she is old, like her own fox fur or the ermine toque or all the other people at the park, and she seems to have misunderstood the way others perceive her. In the end, however, once the young couple have insulted her repeatedly, she does not let on that she's heard them; she simply leaves the park and returns to her home (where it seems that her realization of her real situation in life occurs to her).