In Chapter 4, Diamond gives a number of factors that influenced the move from hunting and gathering to food production. By giving us this list, Diamond challenges the idea that farming would only develop in fertile lands.
One of the major reasons to switch to farming was the decrease in availability of wild foods. This would make it harder to make a living gathering food (as opposed to growing it). It stands to reason that wild foods would remain widely available in fertile areas for a much longer time than in relatively infertile areas. Because the food would run out more quickly in relatively infertile areas, farming would become attractive in those areas before it became attractive to people living in fertile areas.