To answer that question, just think for yourself about how Holden feels about Pencey. If you felt like that, would you want that to be the happiest time of your life?
At Pencey, Holden is completely disillusioned. He hates pretty much everything about the place. He thinks that the people there are phonies and crooks. He is miserable.
So imagine you are being completely miserable at some place and a guy comes up and tells you that his time there was the best of his life. You might be worried that the same is true for you and your life is just going to get worse from here on out.
You see, Holden didn't like Pencey to begin with. In fact, he didn't like any of the prep schools he went to - the teachers were phony, and so were the students. He never applied himself, because he never cared. When the older man was talking about the happiest days of his life being at Pencey, Holden thought it was a load of crap, and was wholly annoyed with the guy. He really didn't like him. Then, to top it off, the guy wanted to go see if his dang initials were still in the bathroom stall when he'd carved them there when HE used to go there. But, see, when he says "it depresses me", and "it kills me", he's not only giving a hint on his case of teenage depression, he's also trying to say, "it really pisses me off" "or it really gets me worked up", just in a different sort of way. (At least, that's what I think.)