The best place to find this answer is in Chapter 10. There, Diamond talks about various barriers to diffusion.
The barrier that Diamond talks about the most is latitude. As we travel north or south, climates tend to change. This makes it hard for crops to be diffused in a north-south direction because they typically cannot go very far before the climate changes too much for them to continue.
But Diamond also talks about togoraphical and ecological barriers. Here, he is talkinga bout things like deserts or mountains or jungles. He gives the example of the US Southeast and Southwest and says that diffusion between those two areas was diffcult because Texas and the southern Great Plains were in between. Those areas were very dry and unsuited to agriculture. This meant crops could not diffuse between the Southeast and the Southwest.
So, latitude is one major barrier to diffusion, but topographical (mountains) and geographical (deserts or jungles) barriers also exist.