Holden immediately begins to regret visiting Mr. Spencer as soon as he enters his teacher's home:
"What made it even more depressing, old Spencer had on this very sad, ratty old bathrobe that he was probably born in or something" (Chapter 2).
Holden is a character who is socially awkward, even in the best of circumstances; the experience of having to be in such a personal space with his under-dressed teacher is extremely uncomfortable for him. He has a difficult time processing the two contrasting perceptions of Mr. Spencer as professional teacher versus sick old man in frumpy bathrobe.
As the visit progresses, Holden also becomes extremely sorry he visited his teacher when Mr. Spencer confronts him about his expulsion from Pencey. He then pulls the ultimate "dirty trick" by pulling out Holden's miserable attempt at his last essay on the Egyptians and reads the entire thing aloud. Holden feels a fantastic "lecture coming on" and sincerely regrets his choice to drop in on Mr. Spencer.