"Racial unconscious" means that each species and culture shares ancient memories, stored in the unconscious part of the mind. Humans and animals alike have memories of their ancestry and those past cultures deeply embedded in our minds, so deep that we aren't aware of them unless something drastic happens to stir those memories.
This happens to Buck when he is kidnapped and taken to the Klondike and must use his animal instincts in order to survive. There are several examples in the book. One is when Buck and the other dogs chase after the rabbit. For the first time, Buck feels the "blood lust, the joy to kill," an instinct he had never needed before. "...and it came to Buck, leading the pack, sounding the old wolf-cry,..."
Another example is when Buck lies by the fire at night and dreams of the man with shorter legs and longer arms, long and matted hair, and a slanted head. "...these sounds and sights of another world would make the hair to rise along his back and stand on end across his shoulders and up his neck."
The longer Buck stayed in the Arctic wilderness, the more his ancient instincts returned. He was not aware of any of these instincts while he was living in the Santa Clara valley because he never needed to use them.