Wallace's job is more important to him than going to the door because he's now an adult and therefore has more important things to attend to than some enchanted garden he used to visit when he was younger.

Wallace desperately wants to revisit the garden, but he realizes that the door to this garden is less important than the metaphorical door that opens onto his career. Wallace is just starting out in life and so can be forgiven for putting away childish things and concentrating on his career. But even so, no matter how hard he's worked—and he's worked incredibly hard over the years—he still can't get the enchanted garden out of his head.

Since setting out on the path of adulthood, Wallace has seen the door four times, though he's never opened it. His life is a social whirl of dinner parties attended by beautiful women and men of distinction. It's no wonder, then, that the charms of the garden he visited as a child should've faded into the distance.