Why does Holden flunk or quit so many schools?
This is a good question.
The short answer is that Holden either flunks out of or gets kicked out of so many schools because he wants to. He's not an idiot, and there is no reason why he can't follow the rules or do the work, so we must assume that either A) Holden wants to be kicked out of schools, either outright or subconsciously, or B) Holden is very indifferent about being kicked out. By nature he is not an indifferent person (he has a lot of ideas and thing to say about the schools and the educations they offer) and so I prefer to think it's "A".
Holden has gone through a very traumatic event with the death of his brother, and this seems to have destabilized him mentally (which is understandable.) He's a young, sensitive guy who doesn't know how to process his feelings about losing his brother, a character that is one of the few in the book that he has an unwaveringly positive view of.
This is open to speculation, of course, but it would seem that Holden has stopped caring or investing himself in his life's progression. Even if his brother had lived it is very likely that Holden would not have "fit in" or enjoyed boarding school, but it also seems like his parents might be using these boarding schools as a way to warehouse a kid that they don't know how to handle. The time period in which the character exists was not "modern" in the handling of a lot of mental health issues, and Holden gets the discipline of school rather than the counseling he needs.
Proof of this seems to be in the way he feels no real remorse at being forced out of yet another school (though I still think it's there, underneath. I remember him up on the hill in the beginning watching like a football game or something, part of the group and yet not.) He still goes to see his favorite teacher, though, showing he craves certain connections with people.
Holden has very little control or power in his life. He is not popular, not well liked, isn't rich, and is not athletic. He's forced into schools he doesn't want to be at with people he doesn't want to be around (perhaps because he doesn't fit in.) He's coping with the death of his brother, who was very important to him, and he's been separated from the other people in his family. Perhaps getting kicked out of school through apathetic behavior is the only way he has left to express himself.