Onions have skins and by "peeling back the onion," one looks beneath the surface of this onion and sees what lies there. So, regarding author's characterizations, readers should examine what lies beneath the surface, so to speak. That is, readers should seek the implications of a character's thoughts, feelings, speech and actions, as well as the reactions of other characters to him/her. For, through the methods of characterization authors enable readers to understand the human condition, thus, conveying conflicts and themes.
In a concise short story of Ernest Hemingway, "A Day's Wait," for instance, "peeling back the onion" with respect to the protagonist leads to the interpretation of this story. The conflict revolves around a misunderstanding of the reading of a thermometer: Unbenown to the father who refers to the boy's temperature in Farenheit, the boy interprets the temperature in Centigrade. Because the father has not "peeled back the onion," he misunderstands the boy's words of "If it bothers you, you can go" and he leaves the boy alone. When he returns, the father finds the boy traumatized because he believes that he is very ill. Fortunately, the father finally understands; however, the boy is never the same because the father failed to "Peel back the onion."
"Peeling back the onion" means to reveal layers, in this case particularly layers of a character's nature or personality. A good, well-developed character will have many aspects, including some behaviors and decisions that may surprise readers. An author who is "peeling back the onion" chooses to reveal only so much about their character at a time so as to let the reader discover more about the complexity of the character.