Biff is bothered by the way that Willy refuses to live in reality. When Biff was in high school, Willy fantasized about Biff's great football future and made excuses for Biff. Even then, Willy was engaged in his dreams more than in the reality of the world around him. Biff can see that Willy does not recognize reality much of the time and he feels this is a choice Willy is making.
Effectively, this refusal to engage in reality means that Biff and Linda are ignored by Willy. They are invisible to him in their reality.
I've also wondered why Biff reacts so strongly to Willy's mumblings. That, as well as Willy's anger toward anything that Linda says, have intrigued me. The latter is easier for me to understand because of my own homelife--a 1950s father feeling trapped by his responsibilities to his family, and every word uttered by his wife or children reminded him of desire to escape. Biff, though, is more interesting. I agree with herappleness that Biff is very much like his father, and the shortcomings of Willy remind him of his own shortcomings. But, lately, when I've overheard my husband talking to himself, I react just as strongly as Biff. The fear that something is happening to one you love provokes a violent reaction. It's not sympathy really, because sympathy implies acknowledgement. It really is anger, outrage. How can this person be letting himself slip this way? How can he do this to me? to himself? Biff must be having these thoughts. His angry responses are feeble attempts to get Willy to snap out of his mumblings, his crazy talk, and they are also his refusal to accept that Willy is declining. How else could Biff desert Willy in the restaurant, leaving Willy in the restroom? I don't think Biff fully lets go of his glorified image of his father until the end of the play.
Biff is no different than his father when it comes to hope for the best and witnessing the worst. He, like his father, would have loved to succeed in life. Like his father, he had hopes for himself in football, in the industry, and he also hoped to reach for the American Dream.
Willy's mumblings were windows into that bittersweet past into which they both had everything at one point, and then their difficulties, Willy's lies, and Biff's desilussion with life, led them to nothing.
It is as if he were watching life destroy itself in front of him one more time, and it is quite painful.. It is not only the annoyance of an insane man going at it and losing all his dignity but also the sadness of relieving everthing all over again.
I think it is that Willy has fallen so far in Biff's opinion since Biff was a teenager, that it is probably that Biff can't take any more reminders of Willy's failings. To Biff, Willy was an idol, larger than life. When Biff caught Willy in his big lie, it was enough to give Biff an excuse to fail at everything. Willy's signs of aging are more reminders to Biff of Willy's imperfection and aging, and by extension, Biff's.
He could possibly think that his father is suffering from some sort of mental illness, after all, the subtitle of the title itself states: "...certain private conversations." It could be that he already knows what is happening to Willy and he doesn't like it.