We really are not told very much about Oliver Bacon's childhood. All that we really know is that A) it was a poor childhood where he had to work hard (and break laws) to get ahead and B) that he and his mother had a decent relationship.
Oliver is described as having been a "slim" and "lissome" boy who would do things like selling stolen dogs in dark alleys. This clearly implies that his family had little money (so does his love for money and status in the story). But we also have to assume he and his mother loved each other. She was sad when he did bad things and he asks her forgiveness towards the end of the story when he makes a really bad business decision.
The Duchess and the Jeweller is a short story by Virginia Woolf. In the story, the duchess is apparently a woman of means and privilege and Oliver prides himself on having been able to rise above his own lowly status to a point where he is in a position to consider and either accept or decline "cards of invitation... from duchesses, countesses, viscountesses and Honorable Ladies." To Oliver this reveals his importance. However, he soon admits that he "began life in a filthy little alley." This fact makes Oliver feel insecure and he "dismantled himself often and became again a little boy in a dark alley..." It is apparent to the reader that his childhood has left Oliver feeling unbalanced and filled with self-doubt. Surrounding himself with wealth is the answer for Oliver in attempting to overcome his vulnerabilities.
The reader also learns more about Oliver's poor and seemingly difficult childhood when he recalls an occasion when "he had been done" or arrested for illegally trying to sell dogs to upper class ladies. Oliver's mother was apparently a significant role model in his childhood, wanting him to better himself. After witnessing his mother's dismay at that time, his life begins to change because Oliver has never been one to miss an opportunity. Knowing where the money is, he starts working "behind a counter... then he had taken a wallet to Amsterdam..." After this he is accepted into a more respected profession, as a jeweler. It seems that Oliver is a fast learner and soon recognizes the advantage of mimicking the older and renowned jewelers, building a reputation for himself. He feels more successful as he acquires expensive properties and possessions.
However, Oliver's worldly success leaves a gap in his life that cannot be filled with his wealth. His need to please his deceased mother's memory also indicates something about his childhood. She may have had high expectations and expected Oliver to acquire wealth rather than foster meaningful relationships. Perhaps she was a domineering woman who thought her relationship with her son should be sufficient for him. He craves love but even that must come at a price which includes overlooking his mother's apparent displeasure. Perhaps he has realized that relationships with people are more fulfilling that relationships with possessions and so is prepared to overlook the duchess's deceit. His mother would not approve but he tries to justify his decision, considering the opportunity to spend a "long week-end" with Diana.