The answer to this question is one word: fear. Mostly, those with Japanese heritage were afraid of racial discrimination and even violence against them simply due to their ancestry. Due to the hardships they all faced, most of them were simply content at keeping their family units together and finding a little bit of happiness in whatever way they could. My very favorite quote that exemplifies this is actually said by Mama:
Mama's first concern now was to keep the family together, and once the war began, she felt safer there [Terminal Island] than isolated racially in Ocean Park.
Therefore, even Jeanne's and Woody's family felt this same fear. The fear was real in that many anti-Japanese groups were being formed by angry and disgruntled white Americans. These groups were designed for everything from committing hate crimes, to taking Japanese American farms away, to making them move to another part of the United States, to actually petitioning for their citizenship to be revoked!
These families, despite their desire to stay, were forced to leave the camps. Some of them tried to return home, but most moved to the Northeast of the United States. Why? This was far away from both Pearl Harbor and from California. It is almost like distance from the central attack locations of the war kept the Japanese Americans safe from more prejudice.
Japanese Americans who refused to leave the camps voluntarily were mostly afraid of attacks. They had experienced racial discrimination even before attacks on Pearl Harbor from sections of the White population within their communities. They knew the situation would be worse because of the attacks and the public opinion against them. Several anti-Japanese groups were formed but most of these groups were formed out of selfish interests. One of these groups included sections of the community such as the White farmers who wanted to take over farms left by the Japanese. Such groups resisted attempts to relocate the Japanese Americans and some even went to the extent of petitioning the government to revoke their citizenship. This level of hostility proved a major challenge during resettlement for the Japanese Americans.
In the end the military remained adamant in their quest to close down the internment camps. Since there was no option to remain in the camps, they left. Some headed back home while most of them decided to settle in different areas in the East, especially New Jersey. They felt these areas were safer because the communities there expressed minimal prejudice against them.
In Farewell to Manzanar, the only choices that were given to the Japanese Americans was to leave their homes in the United States and be relocated to Japan, to be drafted into the US Military, or to leave camp and the West Coast (only if they had a sponsor). Some people refused to join the draft, which gave them only the choice of being relocated. For instance, Papa is called an "inu" for wanting to support the Americans.
In December, 1944, the Supreme Court finally rules that the internment policy surrounding Manzanar is illegal, and that the War Department must close all camps. The remaining residents postpone their departure because of their fear for their futures, leaving their homes and everything they have ever known. In other words, many of the Japanese Americans lived their entire lives on the West Coast. To leave would be abandoning their lives and being forced to start anew in unfamiliar territory. Nevertheless, all are eventually forced to leave.
As far as i van se, they didnt want to show weaknes