The opening stanza is about mourning the loss of childhood, so there is a general sense of melancholy about aging and a heart-wrenching nostalgia/yearning for one's lost youth.  The speaker notes the change from the way he experienced the world as a child and how he experiences the world as an adult.  For him, immortality means the Soul exists before, during and after the life of the body. So, the child is closer to the memory of that spiritual realm than the adult.  Stanza V:

Our birth is but a sleep and a forgetting:/ and later, "not in entire forgetfulness" and "but trailing clouds of glory do we come."

As the child becomes an adult, he is further removed from that dream and becomes mired in a life of "endless imitation." (Stanza VII).  The adult must rely on memory, which fades with age, to get glimpses of that fresh outlook that is characteristic of a child.  The older you get, the more you focus on the day to day repetition of life.  Also, the more you understand of the world, the less miraculous it seems.  So, the adult suffers in that loss of viewing the world with such freshness.  He grieves in "nothing can bring back the hour/Of splendour in the grass and the glory in the flower" (X), but then goes on to say he'll find strength in what remains behind.

The speaker also finds strength or wisdom in that the mature philosophic mind can see "through death" to immortality, and more to your question, as the speaker ages, he learns to more appreciate the fragility of life, the tenderness of the human heart and sensitivity, empathy to human suffering.  So the speaker talks about suffering individually, only being able to intimate immortality from recollections from early childhood.  But he achieves wisdom through philosophical maturity and finds "soothing thoughts" arising from his appreciation of this world, the fragility of mortality and sensitivity to human suffering.