This is because both youth and a flower are beautiful (you can argue) and both of them fade quickly away. So this is a metaphor about how quickly the "good times" end.
You can say youth is a beautiful thing. Young people tend to look better than older people. They are more optimistic. They have their whole lives ahead of them and they do not have the same kinds of worries that the older people do (although they have plenty of their own).
But youth passes quickly (or so you can argue), and all that freshness -- fresh young body and mind -- wilt like a flower.