The reason why Nella didn't return to Denmark is a mystery that can be solved only through reader conjecture.
We learn a lot about Nella through her journals throughout the book. Most importantly, we learn that Nella brought her children to America in order to give them an even better life. The irony is, when Nella could not give them that better life (and, in fact, life becomes even worse than she could have ever dreamed), Nella becomes completely desperate.
How do we know Nella is desperate? We learn the entire story from Rachel's own mouth at the end of the book. Nella is the one who plummets the family to their deaths. Nella pushes her son, Robbie, first. Rachel backs away, so Nella jumps with the new baby in her arms. Finally, Rachel (out of pure desperation) jumps in order to try and cushion Robbie's fall. Rachel, though, is the one that falls on Robbie. Rachel is the sole survivor. Just as it was Nella's worry that moved her from Denmark to the United States, it was Nella's worry that caused her own death and the death of her children.
Apart from the fact that finances also may have prevented a move, it was Nella's desperation that prevented her from moving back to her home country. Someone who is in despair and desperation is not thinking clearly. He or she might not be able to see what would be better or worse for anyone. This is what happens to Nella. Nella feels so very entrenched now in American life (and that her children are victims of American racism), that she sees no other way out other than death. Moving back to Denmark was not an idea that an irrational mind could entertain.