The answer to this can be found in Chapter 16. There, Diamond tells us that linguistic evidence helps to show that one group of people spread south from North China, dominating the people who used to live in the areas into which they spread.
People in all areas of China and the surrounding areas speak Sino-Tibetan languages. By contrast, languages from other families such as Miao-Yao, Austroasiatic, and Tai-Kadai are much more scattered. Diamond says that this means that the speakers of these last three languages used to have "a more nearly continuous distribution." That is, they were not scattered in ancient times. The fact that Sino-Tibetan is so widespread shows that speakers of those languages moved south and displaced speakers of the other languages.