Holden's discussion of "yellow" -- and the two best examples of it -- occur in Chapter 13.
Yellow means cowardly. It is similar to the term "chicken." Holden only uses it in connection with the example he gives about how he would handle it if, hypothetically, he had found out which student stole his gloves at Pence. He imagines himself going to the thief's room, finding the gloves in the thief's galoshes, and confronting the thief verbally. "Okay, how about handing over those gloves?" He then imagines evading the question "Are you calling me a crook?" with "All I know is my gloves were in your goddam galoshes." He would do this again and again rather than start the fight he'd like to start. He could not bring himself to say, "Yes, I am calling you a thief," or to confront the thief more directly than with a cutting, sarcastic comment. He imagines that he would stand there, frozen, for about five minutes, unable to get up the nerve to "sock the guy in the jaw."
Avoiding a direct confrontation is certainly a form of cowardice. So yes, Holden is a bit "yellow" in this imagined scene. Throughout the rest of the book, we see that Holden does not avoid confrontation, but he confronts indirectly. His tendency is to keep prodding or provoking another person, ignoring social signals, until they snap. For example, he keeps provoking Stradlater about whether he had sex with Jane, until Stradlater beats him up. Apparently Holden feels it would be better if he were more direct (although this might mean that he would go around punching people.)
Later in Chapter 13, we see Holden chicken out of having sex with a prostitute who comes to his room. On the face of it, this might look like cowardice. But it is not. It is really sensitivity and compassion. Holden can't bring himself to treat the girl in a dehumanizing way. He is unnerved when she simply pulls off her dress, wanting to have a conversation with her first. He is also depressed by the thought of her life being what it is. Finally, he can't go through with it, and sends her away, but although embarrassing, this is really to his credit. True to form, he does not tell her directly why he has lost interest. Instead he makes up a ridiculous lie about having had a recent operation.