It is notable that Wallace's only experience of actually entering the garden came when he was between five and six years old, and yet he can still recall that experience vividly even decades after the fact. This, in and of itself, illustrates just how much that experience has captivated him across the course of his life.
Wallace recalls his time in the garden as a source of wonder, but at the same time, it also represents a source of deep regret. Remember: for all that his memory of the garden has exerted a powerful hold on Wallace's imagination, Wallace himself is of the opinion that he has actually had several opportunities to return to it throughout his life, but he has consistently set the garden aside, favoring more immediate goals and pursuits. In this respect, one might say that the garden's hold on him is not quite absolute. But even so, these decisions have resulted in a great deal of regret and second-guessing, emotions that seem to have grown over time. Thus, we observe Wallace insisting to the narrator that the next time the door appears, he intends to walk through it. This is a choice he eventually does seem to make, resulting in his death.
Thus, his time in the garden has had a profound impact, shaping the course of his life as well as his emotional perspective on said life (both as a source of wonder and regret).
In H. G. Wells's short story "The Door in the Wall," Wallace—who tells the story of his discovery of a magical garden—provides his audience, Redmond, with more questions, than answers.
When Wallace was a young boy of about five years of age, he wandered off and found himself at a long white wall with a green door in it. It seemed to "call" to him in some way, so that even while the boy tried to fight it, he finally charged through the door before he could change his mind. Inside was a wonderful place where this lonely little boy was able to play, see amazing things (like tamed panthers), and feel the warm welcome of its residents, of which there were many different kinds. His departure from this place was forced upon him as he insisted upon seeing his future in a magical book that held all the pieces—the exact experiences—that had made up his life to that point.
When the young Wallace returned home, bereft of his garden and friends, he was punished first by his father and then by an aunt, for making up stories. Except for a very short time, the garden—and the wall and door—occupied a good deal of his attention. And even though he passed the door several times, there was always a reason for him not to enter through it: a pressing engagement to try to win a scholarship or the possibility of a higher political position, etc.
Throughout his life, then, he spent an enormous amount of time wishing he could return to the garden and regretting that he had so often passed over the opportunity for accomplishments that held little real meaning for him: until he sees his lifetime as years filled with only that regret.
By the end of the story, Wallace is depressed and disappointed. The things that the world uses to measure success (and he is considered by Redmond to be wildly successful), are meaningless to Wallace. In pursuing the measure of success in a man's world, he forgot to seek out the joys of youth, maintaining some semblance of the imaginative child he became upon entering the garden so many years ago.
Ironically, Wallace promises himself that if the door appears to him again, which he highly doubts, he will not pass up another opportunity but will enter; and he assures Redmond that he will never return. The narrator, Redmond, is left to ask himself (and the reader) if in the end, Wallace did not do just that.