Relating to your question, Miss Emily was perceived as an outcast by her town when she refused to pay her taxes. She would not accept the fact that Colonel Sartoris was long dead.
"Miss Emily, “a small, fat woman in black,” met them at the door, and she told them that she had no taxes in Jefferson."
Miss Emily is thought of as an outcast when the townspeople begin to notice a pungent odor emanating from her house.
"It was with the onset of the smell that the townspeople had begun to feel sorry for Miss Emily, as they recalled how Miss Emily’s great-aunt, old lady Wyatt, had gone crazy."
She is considered crazy, like her great-aunt, not warmly embraced by the town, but tolerated.
When Miss Emily starts taking buggy rides with Homer Barron, the townspeople do not approve of a southern woman dating a Yankee. She is made a social outcast.
"When Miss Emily is seen in public with Homer Barron, the townspeople are abhorred on two accounts: first, that Barron is a “Yankee,” and second, that he is a “day laborer,” even if he is a foreman."
Miss Emily, even in death, was a spectacle for the townspeople to stare at and gossip about, speculating on what would be found in her old gloomy house.
".… What other person, or what other house, in the town had ever received this much attention?"
Miss Emily was not an outcast in her town. On the contrary, she isolated herself from the rest of the town, more or less making everyone else an outcast. The very first sentence indicates how curious everyone was about her:
When Miss Emily Grierson died, our whole town went to her funeral: the men through a sort of respectful affection for a fallen monument, the women mostly out of curiosity to see the inside of her house....
Also, the narrator tells us that "Miss Emily had been a tradition, a duty, and a care; a sort of hereditary obligation upon the town." When she began seeing Homer Barron, the townspeople were happy for her, and when they suspected she might use the arsenic to kill herself, they wrote to her cousins to come see her.
If people avoided talking to or having any other contact with Emily it was because she was so cantankerous. She was difficult and set in her ways, and I'll bet people were a little afraid of her!