The basis for such a statement can be found in Chapter 10. There, Diamond tells us that crops can diffuse much more easily from one place to another along the same line of latitude. Thus, we can say that "borrowers" are generally located near to, and at a similar latitude to, the "haves" from whom they borrowed.
To Diamond, the haves are those who developed agriculture. The borrowers did not develop agriculture on their own but rather borrowed it from the haves. This borrowing was much easier if the borrower was at a latitude similar to the have. This is because places of similar latitudes tend to have similar climates. This means that a plant that flourishes in one place is likely to flourish in another place on the same line of latitude.
Therefore, borrowers must be near to the haves (to come in contact with them) and must be at similar latitudes (so the haves' plants can flourish in the borrowers' climate).