Willy's adultery affects the rest of his life and that of his eldest son, Biff.
Until the fateful episode in Boston where Biff finds a woman in his father's room, their relationship was extremely close. Everyone around them noticed the change that occurred. Linda says in Act 1-
LINDA: What happened to the love you had for him? You used to be such pals?
The loss of his son's respect is a severe torture to Willy. His punishment is to endure the knowledge that his actions destroyed his relationship with Biff. He has to keep re-living his mistake through his flashbacks and in the questioning of those around him. Bernard, Charley's son, clearly outlines that Biff's demise started after the visit to Willy. He makes it cruelly clear to Willy that he is responsible for his son's lack of ambition-
BERNARD: ...I've often thought of how strange it was that I knew he'd given up his life. What happened in Boston, Willy?
Biff was devastated to discover that his father was not what he seemed. The knowledge has damaged both of them.