Of the two, Biff was more profoundly affected by discovering his father in an adulterous situation in a Boston hotel room. His confusion turned to understanding and then gave way to tears and deep anger. Ignoring his father's explanations, Biff refused to let his father touch him, and for the first time, but not the last, he called his father a liar. Also, fueling Biff's anger and tears was the knowledge that Willy had given Linda's stockings to the woman. Like Willy, Biff surely had watched his mother mend her own worn-out stockings many times. Throughout the rest of Willy's life, he and his son never discuss what happened in Boston, but Biff's anger did not dissipate. He buried it, but it was always there, below the emotional surface, continuing to poison his relationship with his father.

Willy's dreams did not anger Biff, but Willy's refusal to give them up and face what Biff perceived to be the truth did cause him anger and frustration. This conflict surfaces in the play's conclusion when, as an adult, Biff tries to speak honestly with Willy. Willy's hostility, his refusal to even entertain the notion that he had been wrong about Biff's future and the family's past, sends Biff once again into a screaming argument with his father, their last. After Willy's death, Biff shows no anger toward his father's dreams, but he still maintains, somewhat sadly, that they were wrong.

If psychologists are correct and anger is a symptom of fear or pain, perhaps both reasons for Biff's anger point to the same cause: the pain of his disillusionment when he discovered his father was not the man Biff believed him to be. 

I agree that Biff was much more angry about Willy's futile dreams than he was about the affair.  I think Biff realizes that the affair was a manifestation of Willy's futile dreams which he likely would never have had if he were more level-headed and accepting of his lot in life.