If you think about Tom Robinson and his family, you can see how racism destroyed him and his family. Even though Tom was obviously innocent, and a much, much better man than Bob Ewell, because of racism, Tom was given a guilty conviction. This eventually led to his death, and to his entire family being left destitute. If it wasn't for the racism that still reigned in the hearts of many people in Maycomb at the time, Tom would have gotten off free-as he should have. Instead, racism allowed a cruel, evil man like Bob Ewell, who framed an innocent black man for rape just to cover the fact that he beats his children, walk free. Racism fueled Bob's accusation, and even his attack on the children, because his pride was so hurt. Bob was ashamed to have been proved the inferior in character to a black man, and so he vowed revenge, which led to his attack on the children. This didn't ruin the children's lives, thanks to Boo's heroic rescue, but it did impact them for the rest of their lives. Jem and Atticus are greatly upset by the guilty verdict, which shows how racism can utterly upset someone who is confronted with its cruelty.
So yes, racism did destroy the lives of people in the novel; mostly black people, but there were resounding impacts on white people in the novel too.