Biff knew that his father would question whether he had actually gone to see Bill Oliver to ask for a loan. His impulse to take the expensive pen may have been motivated consciously or unconsciously by a desire to have tangible proof that he was in Oliver's private office. But after he had stolen the pen he quickly realized it didn't prove anything positive but only that he was a thief. 

Another possibility is that the pen seemed like a trophy. Biff's best years were when he was a star athlete in high school. He was always bringing home trophies, and Willy displayed them all with pride. Biff may have had an impulse to bring his father the pen as another trophy. But it would have been a poor substitute for the ten- or fifteen-thousand dollars that Willy expected his son to borrow from Bill Oliver to start a sporting-goods business. The pen was not something to be proud of but, on the contrary, something to be ashamed of. He couldn't give it to his father. He couldn't keep it himself because it would remind him of his humiliation. He couldn't throw it away. And he could hardly return it to Bill Oliver, who would already be missing his expensive pen and realizing what must have happened to it. 

The pen makes Biff realize his mediocrity. He doesn't deserve to own such a pen. He regrets taking it, just as he regrets trying to be something he cannot be. 

Biff Loman resembles Christian Darling, the focal character of Irwin Shaw's story "The Eighty-Yard Run" (1955). See reference link below.