As with many of Stevenson's verses for children, “The Gardener” is based upon an intergenerational clash. The child speaker in the poem just wants to play, but the gardener, an old man, isn't interested, much to the speaker's puzzlement and annoyance.
The gardener takes his gardening very seriously indeed. In fact, he's described by the speaker as “old and serious,” an indication that gardening is more than just a hobby for him. Unsurprisingly, the gardener has no time for playing games or doing anything else for that matter. He's too busy digging flowers, cutting hay, and doing all the hard work that needs to be done in his garden.
The gardener evidently takes great pride in his work, as we can see from the fact that he makes the speaker keep to the gravel path. The last thing he wants is for some overexuberant child to walk all over his garden, trampling his lovely flowers underfoot. For good measure, when he's done for the day, he puts his tools away, locks the door of his shed, and takes the key with him.
The old man's evidently worried that if he doesn't take these precautions, the speaker—or perhaps another child—will cause mischief in his beloved garden.