The Color of Water by James McBride is both the autobiographical story of James's search for his heritage and his mother's memoirs of growing up a white, Jewish girl in an abusive family. It is clear that his mother's secretiveness about her past contributes to many of the troubles James has in his life; it is equally clear that he grew up in a family everyone would consider to be unconventional. Because of these two things, people had different impressions of James throughout his life.
James was the youngest of eight children, and his father died when James was born. Eventually he would become the oldest of the youngest five siblings. His life was eccentric and the family's neighbors and friends sensed this. James's mother was clearly a white-skinned woman who lived as if she were a dark-skinned Negro. She was adamant about getting her children the best education possible, sending them to Jewish schools while fostering all the worst stereotypes about Jews.
The McBride children are successful in these white schools, but there is certainly some sense of pity for them. James comes home one day (chapter 10) and asks his mother what a "tragic mulatto" is; obviously someone called him that and he did not understand what it meant. This is a clear picture of how others perceived James and perhaps his siblings: tragic because they do not have a clearly defined place in society and mulatto, of course, because they are mixed-race children.
When James gets in trouble later in high school, he is sent to spend the summer with his sister in Kentucky. James is a troubled teenager and begins hanging out with unsavory characters. When he tries to ask one of them for some advice (in chapter 14), the jobless man tells him just to "forget it," implying that James is not even worthy of his time--the one thing the man has plenty of.
Later, James turns his life around and becomes whole as he discovers his mother's heritage, information which helps him make sense of his unanswered questions and finally places him in a more "normal" and accepted position in society.