The answer to this is found on p. 129 in my book. Basically, the answer is that bitterness in almonds is controlled by only one gene. In oak trees, by contrast, the bitterness of acorns is controlled by many genes.
The reason that this is important is because this means that about half of the almonds from a tree that is not bitter will also produce trees that are not bitter. By contrast, almost all of the acorns from a "good" oak tree would (if you plant them) produce an oak tree whose acorns are bitter.
This means that it would be much easier to selectively breed good almonds than it would be to breed good oaks. Therefore, people would quit trying to domesticate oaks but would continue to try to domesticate almonds -- they would have a much better chance of success with the almonds.