Diamond gets his evidence for this connection from two (or three, depending on how you want to look at it) types of sources. These include historical anecdotes, personal experience (which can be seen as a kind of anecdote) and studies done by cultural anthropologists.
We can see an example of historical anecdotes in Chapters 2 and 4. In Chapter 2, Diamond gives anecdotal evidence of how Polynesian societies that farmed were more developed in a number of ways than those that did not. In Chapter 4, he discusses (for example, on pages 90 and 91 of the paperback edition of the book) how farming led some societies to develop in economic and military terms. Sometimes, Diamond uses anecdotes from things that he has seen personally. An example of this can be found beginning on p. 269. There, Diamond recounts his experiences with the Fayu of New Guinea, who were hunter-gatherers and lived in very primitive societies.
However, Diamond also uses more systematic sources. The major place where this can be seen is in Chapter 14. There, Diamond relies on a typography of societies that has been developed by cultural anthropologists. He suggests a number of studies by such scholars as further reading in the back of the book. He uses these studies and the typology to show that more complex political and social institutions arise when societies have farming.
Thus, Diamond uses a number of different sources of evidence for these claims.