We are never exactly told why someone would consider a boys' home to be worse than jail, so I can only speculate.
I would think that a boys' home might be worse than jail because it would not have the same kind of tough image that going to real jail would have. Someone who got sent to a boys' home would lose their family and such but not get the "benefit" of seeming tough and cool.
So it's an issue of seeming cool. Jail is tough, being sent to a boys' home sort of implies that you're a kid.
Boys home's during the era when the boys were adolescents during the story in the book The Outsiderswere not very good places. They were not much more than jails. The boys were incarcerated but still had to attend public school so there was the humiliation of being in the home, being without one's family, and for the Greasers it meant being away from their community and close group of friends.
This is not explained in the book, but having a brother was placed in such a home during the era gave me some insight into the lifestyle of the boys. I agree with the previous editor about the reason that it would be bad. It would be lonely and humiliating and would not have the coolness having been in jail would have had.