The answer to this question can be found in Chapter 7 of Guns, Germs and Steel. One reason is that almond trees grow much quicker than oaks, which makes them better for agriculture. Also, oaks rely on squirrels to reproduce. They gather and bury acorns faster than humans can harvest them. Some of the acorns are not eaten, and grow into trees. Because of this, oaks have evolved acorns that are small enough for squirrels to eat, which means that people would have to harvest many more to make it worth their while in terms of calories. Almonds are bigger, and thus provide more nutrition. The last reason is that both acorns and almonds are bitter. However, genetic variations do occur, but they occur in such a way that favors almonds for human cultivation. In almonds, bitterness is determined by just one gene, while in acorns, bitter flavors is controlled by many separate genes, all of which would have to be bred out to assure tasty acorns. For these reasons, it was much easier and efficient to develop almonds for agriculture. Acorns were generally used by hunter-gatherers to supplement their diets.
Source: Diamond, Guns, Germs, and Steel, 129.