This is a difficult question to answer simply because it is a multi-generational story which covers lots of time and places. The Color of Water was written by James McBride, and it is an autobiographical journey both of his life and his mother's life.
James McBride is one of twelve children and grows up in Brooklyn, New York in the sixties and seventies. He spends most of his life through adolescence either there or in Queens. His mother moves them to Delaware during his high school years. To be a bit more specific, though, he does not generally attend school in Brooklyn or Queens when he lives there. His mother sends him to whatever school she can in order to improve his chances at a superior education. He attends Oberlin College.
Ruth McBride, the author's mother, was born in Poland and lives in Manhattan when her family comes to America. Her father is an itinerant rabbi, so the family moves around a lot when Ruth is young. When she is eight or nine, her family settles in Suffolk, Virginia. Later she leaves and goes to Manhattan and finally ends up in Brooklyn, Queens, and Delaware, as outlined above. Her husband's family is from the South, so she does spend some time there.
When James begins to trace his mother's history, he, too, spends some time in Suffolk.
Like so many of us, the McBrides spend their time in many places over the years.