Though Willy thinks of his sons' pasts as idyllic, the reality is otherwise. For example, near the beginning of the play, Bernard tells Biff that Mr. Birnbaum is going to flunk Biff in math unless Biff starts applying himself. Bernard is allowing Biff to cheat off him, but that doesn't help Biff pass his final exams. When Mr. Birnbaum winds up flunking him, Biff doesn't graduate but can make up his missing credits in summer school. However, for reasons no one can understand, Biff never attends summer school.
Biff also routinely steals objects from Bill Oliver's store. Years ago he stole several balls, and later he steals a pen and is too obstinate to return it and admit he took it (apparently by mistake). Linda also tells Willy about Biff, "He’s too rough with the girls, Willy. All the mothers are afraid of him!" Biff clearly doesn't treat women well, and he isn't a very nice person. Happy, for his part, is always in Biff's shadow, and he is acutely aware that his father loves Biff best. In short, the sons do not experience the idyllic childhood that their father later remembers.
There is also the moment when Biff's own head clears and he realizes that he has allowed Willy's idyllic vision of the past to skew his own. This occurs once Biff goes to his job interview and steals a pen. He later confronts Happy about it, becoming angry because he wants to know how he or Willy had ever gotten it into their heads that he had been a salesman. He now remembers that he was merely a stockboy.
You might have hints of this less than ideal past in the fact that Biff has not made anything of his life since that time. It makes one wonder, hmm, was the past as good as it seemed? There were, however, some more specific points. In Act I, part 2, Biff took a football from school without permission, a sign of irresponsibility (or worse). However, it is in part 3 of that act that the signs really pile up. Biff wants Bernard to cheat for him (definitely a bad sign), he's too rough when interacting with girls (a worse sign), and Linda ends up crying.