Atticus was inferring to the fate of "Boo" Radley, first isolated and ostracized from society by his father, and then (just when there was a possibility of a new life -when his father died!) by his older brother Nathan. Boo's father had struck a deal with the sheriff to keep his son "out of sight, out of mind" if he would not be sent off to a technical school as part of a reform program.
"Boo" was not a harderned criminal but only an adolescent guilty of boyish pranks, and the other boys who had also locked up the school official in the outhouse and "stolen" a car to go joyriding actually profitted from the program. They got an education, got jobs, and got on with their lives. "Boo," however, was kept inside the Radley house and so "spooked" by his father and older brother that he never did come out in public again. In such a way he became phantom-like and only a shadow of his former self.