To answer this question, let us consider what this poem is about, and more importantly, how it criticises establishes religion. Notice how the poem describes a visit by the speaker to "The Garden of Love," which now has a "chapel" built in the midst of it, where the speaker as a child used to play. We can infer therefore that the "The Garden of Love" was a happy, carefree place of enjoyment and pleasure. However, now, the buidling of the chapel seems to have converted it into a place devoid of pleasure. Instead, on the chapel there is a sign saying "Thou shalt not." Turning away, instead of the flowers that the speaker remembers in this Garden, all he sees are "tomb-stones." The last two lines in particular convey the criticism of Blake of established religion:
And priests in black gowns were walking their rounds
And binding with briars my joys and desires.
The overwhleming force of this poem is the way in which religion, symbolised in the "chapel," and its ministers, symbolised in the "priests in black gowns," choke our "joys and desires" with their religion and creed and turn joy into ashes. We can understand therefore why such overt criticism of the church would have been controversial in its day.