Of course, the illegitimacy of both Sibyl and James makes it extremely unlikely for them to marry well, which gives Mrs. Vane another reason to keep the nature of her relationship with their father a secret, as it could impact on the chances of Sibyl marrying Dorian. It is important not to underestimate the importance of how the stigma of being illegitimate impacted people during Victorian times.

In addition to the above, Mrs. Vane's biggest agony--and secret--is that she was not married to Sybil's and James's father. This is why they live in poverty--he left without leaving them and provision. Also, she fears the ostracism and rejection that unwed mothers and their children suffered in their society.

I hope  you still need an answer. I found a possible one, but if this is contrary to what you learned, please let me know what info you got on your side:

Apparently Mrs. Vane used to be an actress, and her former husband either left her or died. She seems, however, to carry some form of shame each time she and Sybil speak, as none of them come clear about why the mother "should be" sad.  Now, she and her daughter have to live in that East End cheap. There is also something about James. He seems to have a past as well as Mrs. Vane and now he is going to Australia to either hide something or start over. We know for sure that he is not the model child, and Mrs. Vane is very vocal about the way she feels about people, and her superficiality with understanding that Dorian is good simply because he is an aristocrat.