Voltaire's story is largely an indictment of claims that there are forces that presume a power higher than human intellect or reason. As one of the Enlightenment philosophies, this makes sense.
Multiple such forces lead to sorrow and chaos. First is, perhaps, human biology. Candide is drawn to Cunegonde, whose name may be a sexual pun. From this early desire—the acting on of which leads to Candide's first fall and expulsion from the garden of Westphalia—sexual impulses (Candide's as well as other's) lead to human disease and violence.
More damaging, though, are the impulses to religion. Voltaire was especially contemptuous of religions that said they alone were the "chosen" or "only" inheritors of God's blessings. We see this in the ridicule of Jews and Catholics, whose claims for superiority cause them to harm others. Only the Anabaptist Jacques, who risks himself for others regardless of their religion, is presented as honorable in his religion.
The Thirty Years' War, linked to Ferdinand II's imposition of Catholicism on the Holy Roman Empire's inhabitants, illustrates a nightmarish view of religious zeal that spreads into other areas of public life and fosters increased nationalistic zeal as well.
Such absolute and passionate commitments to abstract ideals impedes a rational consideration of how one is meant to live and how one can best pursue one's happiness. "Cultivating one's garden" becomes both a tangible and a metaphorical message on how to avoid the irrational and how to dedicate oneself to concrete good for oneself and ultimately others around oneself. In this way, free will can exert itself on a meaningful object without harm to others equally seeking to cultivate their garden.