A subjective experience of race, in my view, refers to what an individual's personal experience is regarding race. In Nella Larsen's novels, race is experienced in different ways by different characters. Larsen's characters in both of her major novels, Passing and Quicksand, are conflicted with regard to race because they are of mixed race. Therefore, their subjective experience of race is also conflicted. What does this mean? They are confused and they don't know who they are. They are not comfortable in either black or white society. They try to live in black society, but they find they do not fit in there. They try to live in white society and they find they do not fit in there.
I have become convinced in reading Larsen's works that many people find the issue of race confusing because we have not personally (or subjectively) experienced what she experienced. Larsen herself was of mixed race, and the confusion and lack of place she experienced in her life is mirrored in her fiction. She is often referred to as being a "liminal" writer with regard to race. A liminal state of being is characterized by ambiguity and confusion or disorientation. This comes through in her writing and for readers that cannot personally relate to this situation, her novels are unsettling and her characters are hard to figure out apart from the ambiguity.
In Passing, for example, Clare lives with two white aunts who pass her off as white (hence the title). Clare is part black. Clare meets up with her childhood friend, Irene later in life. Clare has married a white bigot who does not know she is part black. Irene has married a black doctor. The two women re-establish their friendship, each enamored with the other's lifestyle. Irene begins to suspect that her husband is having an affair with Clare. But is he? It is ambiguous. We are never certain. Clare's husband finds out about her race at the end of the novel. Clare mysteriously dies after falling out of a window. Did she commit suicide? Did someone push her? If so, who? Was it her husband? Was it Irene?
In this novel, race (as well as class, and gender) is defined by the subjective experience of the characters. The typical ideas of race are very blurred in Clare because her subjective experience is blurred.
In her other novel, Quicksand, the main character, Helga, is also a mulatto. She, too, tries to live in both black and white society, never finding happiness in either one. She is conflicted when she lives among blacks, so she moves to Denmark. While in Denmark, however, she misses the company of black people. A white man proposes to her and she tells him she can never marry a white man.
I have recently discussed these novels with some colleagues that are black and they tell me I cannot possibly "get" Larsen's writing because I am white. I don't believe this is true, but their comments relate to your question about subjective experience.
Read about Larsen here on enotes.