This poem falls very strongly into the pastoral tradition of literature, which presented nature and in this case, the garden, as being a haven from the troubles and strife of the messy world in which we live. Having established this, let us turn to stanzas three and four to answer your question. Having described the garden as the dwelling place of Quiet and Innocence, who are personified by Marvell, he compares this idyllic calm and tranquility to the "Fond lovers" who are subject to "passion's heat," which contrasts directly with the calm and peace that the garden stands for. Marvell goes on to refer directly to famous mythological couples who were subject to the ravages of love to make his point:
Apollo hunted Daphne so,
Only that she might laurel grow,
And Pan did after Syrinx speed,
Not as a nymph, but for a reed.
Marvell states that when the rigours of passion have run their course, every lover needs to end up in the garden to relax and calm down, and indeed the two stories he refers to emphasises this. Classical myth is therefore used in this instance to represent the importance and the vital nature of the garden. Humans, after experiencing the more visceral and exhausting emotions that we are capable of, desperately need to return to the garden in order to recharge our batteries and experience true calm.