At the time of the novel, life in Harlem, New York, is very difficult for African Americans. The title of the novel, "Passing," refers to lighter-skinned black people being able to "pass" for white people, or at least for predominately white mixed-race individuals.

Irene, in the story, is relatively light-skinned, and so she can "pass" as white, or nearly so. Brian, however, is much darker-skinned and has no chance at all of passing as white. He endures severe racism and has to deal with prejudice constantly, which his wife is able to avoid.

In Brazil, race relations are much less severe, and he would not have to deal with quite as much prejudice. Additionally, there are plenty of darker-skinned native Brazilians, so Brian would potentially be able to blend in there much better than in America.

Brian wants to go to Brazil to escape the deep levels of racism prevalent in American society. Unlike his wife Irene, Brian is dark-skinned and so cannot pass for white. He figures that as Brazil is more of a multi-racial country than the United States, he'll find acceptance there. Brian has a very high-status job—as a surgeon, no less—and yet still doesn't receive the appropriate level of respect from white society. Perhaps in Brazil things would be different.

One can only surmise because there's an element of fantasy about Brian's wanting to relocate to South America. One gets the impression that he hasn't really thought this thing through; his desire to head off to Brazil seems little more than a whim. It's this romantic streak in Brian, among other things, that makes Irene suspect that he's having an affair with Clare.