For one thing, Holden has developed an automatic aversion to all adults. He notes that the old man was probably a good guy, but thinks he is a phony nonetheless. Holden is so critical of everything and everyone around him that when Phoebe asks him what he likes, he has trouble thinking of anything. He finally says that he likes Allie and talking with her (Phoebe). But what he actually thinks of are the nuns and James Castle. Holden admired Jame's Castle's defiance. Holden tries to embody this kind of defiance in nearly everything he does. That's why he's got a chip on his shoulder about adults, phonies, and Pencey in general. When the old man says his days at Pencey were the happiest of his life, this depresses Holden because his own days at Pencey were miserable. It's hard to be happy when you're critical of everything around you.
We see the duplicity of Holden in this chapter (22). Despite illustrating his general dislike of everyone who isn't Allie or Phoebe, he expresses a desire to be a protector: a catcher in the rye.
What I have to do, I have to catch everybody if they start to go over the cliff--I mean if they're running and they don't look where they're going I have to come out from somewhere and catch them.
Holden wants to protect children from the phonies of the world, but to do this he would have to accept that these children (who will become adults) are worth protecting. Perhaps another reason the old man depresses him is because he envisions other children having miserable experiences at school similar to his (Holden's) own. Holden is still too stubborn to admit that he could get anything of value out of Pencey. So, he condemns the old man as being basically good but depressing.