One can find a couple different quotes, in Harper Lee's novel To Kill a Mockingbird, showing Boo Radley as an outcast.
Inside the house lived a malevolent phantom.
Here, one's understanding of what malevolent means supports Boo's state as being one of an outcast. Malevolent means willing to do evil. The children, Scout and Jem, "know" Boo to be a "malevolent phantom." This supports the fact that his is seen as an outcast based upon the fact that one who would wish to do harm to others, while remaining hidden, is reason enough to support his role as an outcast.
At one point in Maycomb's history, some of the town's pets and chickens were found murdered and mutilated. Although another resident was found to be guilty of the crime, the people of Maycomb
still looked at the Radley Place, unwilling to discard their initial suspicions.
This was another reason that Boo was cast out by the town.