In this section of The Catcher in the Rye, Holden is returning from the bar where he had listened to Ernie play the piano, a bar full of prep. school and college students. Holden chooses to walk back to the hotel, saying that he did not feel like getting in and out of another cab.
While walking back to his hotel, he thinks about how "yellow" he is. In thinking about his cowardice, Holden creates a hypothetical scenario to demonstrate what he would do in a situation that would call on him to be bold.
The situation he imagines features a pair of stolen gloves. Holden imagines a scene where another student has stolen a pair of his gloves. When Holden confronts the student, he feels that he should punch the thief in the jaw. He imagines that the thief will stand up to him and confront Holden with the question, "Are you calling me a thief?" Holden pictures himself demuring and choosing not to punch the student and not to direclty call him a thief.
Holden claims that he hates fist fights because he cannot stand looking at the other person's face during a fight.