There are several reasons that Boo is considered an outsider--and the town's most talked-about citizen. First, Maycomb is a small town where everyone knows everyone, and the Radleys never quite fit in.
The Radleys, welcome anywhere in town, kept to themselves, a predilection unforgivable in Maycomb. (Chapter 1)
The family worshipped in their own home instead of embracing the social aspects of church; they rarely socialized with anyone (old Mr. Radley always failed to respond to Jem's and Scout's greetings); and they kept their
... shutters and doors... closed on Sundays, another thing alien to Maycomb's ways. (Chapter 1)
The family was already outcasts before Arthur Jr.'s troubles, but when he was restricted by his father's unusual form of house arrest within the Radley House walls, the town made "Boo" its favorite subject of gossip. When both Mr. and Mrs. Radley died and Boo did not come out--instead, being looked after by his brother, Nathan--the gossip only increased. In Maycomb, people gathered in front yards and porches to discuss the day's events, but Nathan kept to himself and Boo was never seen. Boo was accused--unfairly and without substantiation--of
Any stealthy small crimes committed in Maycomb... People still looked at the Radley Place, unwilling to discard their initial suspicions. (Chapter 1)
Boo was seen but once in nearly two decades--after he was again incarcerated for stabbing his father's leg with a pair of scissors--and he again retreated into the seclusion of the family home, never to be seen until he reappeared on Halloween night to rescue Jem and Scout. Boo's status as Maycomb's leading outcast was solidified by his parents' prior actions; his own run-ins with the law; and the fact that he was never seen, even though everyone in town knew he was still alive since, as Miss Maudie told Scout,
"I know he's alive, Jean Louise, because I haven't seen him carried out yet." (Chapter 5)