Diamond addresses this question in Chapter Seven. Oaks, unlike almond trees, grow very slowly, which is obviously not good for agriculture. Second, oaks spread largely through the efforts of squirrels. They are so industrious in gathering and burying acorns that people couldn't really compete with them, and they have been such an effective method of spreading oaks that oaks have evolved squirrel-sized, rather than people-sized fruit in the form of acorns. Almonds are bigger, and provide more nutrition. Finally, both acorns and almonds are naturally bitter with some variations. However, bitterness in almonds is controlled by only one gene, while in acorns, it is controlled by many genes. This meant that it was easier for agriculturalists to cultivate non-bitter varieties of almonds.
Source: Diamond, Guns, Germs, and Steel, 129.
this is due to the number of genes that enables almonds to be bitter...almonds-1 acorns-3