The chapter you need to look at to answer this question is the one intriguingly entitled "How to make an Almond." This fascinating chapter discusses the way that different foods were able to be domesticated and why, and then also looks at the many other kinds of foods that were not able to be domesticated. Even though Native Americans used acorns extensively in the eastern United States, acorns have never been domesticated in the same way as almonds have, when both are naturally bitter. This is for three main reasons:
1) Oak trees grow very slowly, compared to an almond tree that grows in three or four years. An acorn needs about a decade.
2) Squirrels eat most of the acorns from oak trees. It is difficult for humans to gain access to a wide range of acorns to selects the ones they want to use.
3) Finally, the bitterness in almonds is controlled by one single gene. In oak trees, the bitterness is controlled by many genes. Note the logical conclusion that this would have:
If ancient farmers planted almonds or acorns from the occasional nonbitter mutant tree, the laws of genetics dictate that half of the nuts from the resulting tree growing up would also be nonbitter in the case of almonds, but almost all would still be bitter in the case of oaks. That alone would kill the enthusiasm of any would-be acorn father who had defeated the squirrels and remained patient.
So we can see that oak trees and almond trees are very different in lots of ways, regarding the time it takes for them to grow, the way that squirrels consume the majority of acorns, and then finally the genetic story behind their bitterness. This meant that it was a lot easier for almonds to be domesticated whereas oak trees have not been domesticated.