Oliver looks at his past with both satisfaction and disdain. He's proud of his meteoric rise but also ashamed of his beginnings; he's aware that he'll never be as highly regarded as he would be with a less questionable background.
At the beginning of the story, Virginia Woolf writes:
"Behold Oliver," he would say, addressing himself. "You who began life in a filthy little alley, you who…" and he would look down at his legs, so shapely in their perfect trousers; at his boots; at his spats. They were all shapely, shining; cut from the best cloth by the best scissors in Savile Row. But he dismantled himself often and became again a little boy in a dark alley.
He starts the statement by admiring himself for the trapping of his wealth and success—his fine clothes. However, he picks himself apart and mentally pushes back to being a little boy who didn't have many choices or much opportunity. Even though he's a different person and outgrew that place, he still can't quite escape it mentally. He yearns for a higher social class.
That's why at the end of the story, he makes a bad decision and trades lots of money for fake pearls. When the chance to spend time with the high-born girl he loves is dangled in front of him, he goes against the common sense that helped him rise up from the streets, gives the money to the Duchess, and soon finds out he was fooled.