The appearance of daffodils is often symbolic of heralding spring, which is a time of new birth and beauty. The speaker in "To Daffodils" examines how quickly this beauty fades and "weeps" to see the glory of these beautiful flowers fade away so quickly. In some areas, daffodils only bloom for about six weeks of the year; thus, the promise of their beauty is fleeting—a whisper in time.
The speaker then notes that "we" humans have comparatively just as short a time to spend on Earth as the beautiful daffodils. Humans, like the beautiful flowers, grow quickly only to speed toward their own decay. Each hour that ticks by brings everything in nature one hour closer to death.
The speaker then notes other beautiful things in nature which fade quickly: summer rains and the dew of morning. Within a quick time, both dry up and are erased. The speaker notes again that "we" also dry away quickly and are speeding toward a time when we will "Ne'er ... be found again."
The image of time, therefore, is fleeting. The message is quite reminiscent of Robert Frost's "Nothing Gold Can Stay," noting that the most beautiful parts of nature never last long. As humans are part of nature, too, they face the same hastening progression toward an eventual decay, passing from Earth and never to be seen again. Time on Earth is extraordinarily brief.