There are two main reasons that Holden gives as to why he is "sort of sorry" for visiting Mr. Spencer. The first reason is because Mr. Spencer has been sick, and Holden has to visit him in his room.
"He was reading the Atlantic Monthly, and there were pills and medicine all over the place, and everything smelled like Vicks Nose Drops. It was pretty depressing. I'm not too crazy about sick people, anyway. What made it even more depressing, old Spencer had on this very sad, ratty old bathrobe that he was probably born in or something. I don't much like to see old guys in their pajamas and bathrobes anyway. Their bumpy old chests are always showing. And their legs...always look so white and unhairy".
It is uncomfortable for Holden to see his history teacher looking sick and old. He is used to interacting with him in the more professional environment of the classroom. To see his teacher as vulnerable, elderly, and mired in all the unpleasantness of illness is hard for Holden to take.
After the two have engaged in a bit of small talk, Holden begins to "feel a terrific lecture coming on", which is the second reason why he regrets having come to visit Mr. Spencer. Mr. Spencer knows that Holden has been kicked out of Pencey, and begins to repeat the same admonitions Holden had already received from Dr. Thurmer, about "Life being a game and all...and how you should play it according to the rules". Mr. Spencer than brings up Holden's parents, what "grand people" they are, and how they will react when they hear the news of their son's expulsion, before proceeding to the topic of the reasons why Holden has been expelled in the first place. He actually confronts Holden with a sample of his poor work, which embarrasses Holden and makes him feel resentful.
Mr. Spencer ends his lecture by telling Holden that he'd "like to put some sense in that head of (his)", and that he is only trying to help him. When Holden finally leaves, he is irritated, focusing on the "phoniness" of Mr. Spencer's sincere concern, and he feels "sad as hell" (Chapter 2).
J. D. Salinger probably felt that he should not have Holden leave Pencey without showing at least one of his teachers. The description of the private boarding school would seem incomplete if Holden only included a few of the students and none of the faculty. Salinger probably chose to set the meeting between Holden and Mr. Spencer in the teacher's bedroom because that would be a way to have a one-on-one conversation and not get involved with a variety of schoolteachers. Mr. Spencer might be said to represent the entire faculty. This may have led to Salinger's making Mr. Spencer sick with the grippe, which would further suggest the funny details of the old man's appearance in his bathrobe and the smell of Vicks Nose Drops.
Holden is lonely. His loneliness is felt throughout the novel. His loneliness is the driving force that moves him through all his meetings and mini-adventures. He goes to see his old teacher in the hope of forgetting his loneliness for a little while--but Mr. Spencer is in a different dimension. As Holden says,
You don't have to think too hard to talk to a teacher.
Holden has had plenty of experience in flunking courses at several different schools. He has learned that teachers don't like to flunk students because it makes the teachers look bad. The obvious question parents and administrators will want to ask is: "How could you have had this student in your class for a full semester and not have motivated him enough to get a better grade than an F?" This may be especially true with teachers at elitist prep schools where the parents are paying a lot of money to have their sons educated. If they wanted their kids to get straight F's, why not send them to a public school? Mr. Spencer is more concerned about his own shortcomings, and more about his own feelings, than Holden's. He asks, defensively:
"Do you blame me for flunking you, boy?"
Holden ends up consoling the teacher rather than the other way around. He says:
"No, sir! I certainly don't.
"What would you have done in my place?" he said. "Tell the truth, boy."
Holden gets nothing out of the meeting and comes away feeling worse than he did before he went there. But this is the sort of thing that happens to him throughout most of the novel. His meetings with people in New York are usually disappointing--and in a couple of cases, as with Maurice and with Mr. Antolini--they are worse than disappointing. Holden not only felt "sort of sorry" for visiting Mr. Spencer, but he probably felt "sort of sorry" for meeting up with Sally Hayes and some of the other people he contacted in Manhattan.
mr spencer is old