Mattie's comment is significant in a couple of ways. First, people who must leave their former lives quickly and unexpectedly do not have time to collect all of their literal paperwork--proof of identity, financial documents, etc.--that might be necessary to start a new life in a different setting.
More importantly, Mattie stresses the upheaval caused in people's lives when they must flee all that they know. Not only do they leave behind humans with whom they have connected or at least been influenced by, but they often must leave behind memories and important items that represent who they once were. Yes, it would be good not to have any "skeletons in one's closet"--at least not close at hand--but along with that lack of a "file cabinet" comes a sense of isolation and identity confusion.