The answer to this can be found in Chapter 16. Specifically, it can be found on p. 331 in the paperback edition of the book. There, we are told that
…ethnic Chinese already tended then… to feel culturally superior to non-Chinese “barbarians,” while North Chinese tended to regard even South Chinese are barbarians.
Diamond discusses this fact within the context of his larger discussion of how China came to be unified. He mentions that it was relatively easy for things that were invented in one part of China to diffuse to the other parts of China, even along a north-south axis. He mentions that many of the things that made Chinese civilization were originated in North China. He mentions that writing was invented in the north and moved to the south. So did bronze technology and the creation of actual states. What this all meant is that the north was generally more advanced than the south. For that reason, the people of the north tended to regard the people of the south as barbarians.