The speaker is commenting upon what he sees in the last days of autumn. The use of the adjective "late" in the title might indicate that it is late in the year as autumn approaches winter. The field has been mowed and all that is left of the garden are withered weeds, meaning the harvest has already occurred. This is more than just evidence that it is late autumn; it also suggests death and grief. Thus, "late" also refers to being old or late in life.
In the third stanza, the speaker watches the last brown leaf fall from a solitary tree. This is a melancholy image which clearly symbolizes death as well as the upcoming winter. In the final stanza, the speaker finds himself where he began, suggesting the cycle of life. He finds the last "faded" aster (flower) and promises to give it to an unspecified person. Perhaps he is taking the flower to someone's grave, but it is just as likely that the speaker is near death himself. In this case, the fading flower represents the speaker's own fading life as he prepares to be reunited with his lost loved one.
The end of things is a theme in this poem. Autumn signals the end of the year in nature. This is paralleled by the speaker's death and/ or the death of his unspecified "you" in the last line. Note that "you" might still be alive. If this is the case, the speaker endeavors to deliver one last flower before the speaker fades from this life. However a reader might interpret these scenarios, the idea of being reunited brightens an otherwise grim poem.