This line appears in "My Heart Leaps Up" and "Ode: Intimations of Immortality" or its longer title, "Ode: Intimations of Immortality from Recollections of Early Childhood."
In "My Heart Leaps Up," Wordsworth expresses his hope to hold onto, or experience as new, the glory and freshness of childhood experience. As we age, things become familiar and therefore, less miraculous. The lesson the Child teaches man (the child's older self) is to keep experiencing life with this appreciative and transformative attitude. In this context and in the context of the "Ode," "man" can refer to the child's older self but also mankind, or all of humanity.
Thinking in terms of time, I, as a child, exist before I exist as an adult. Therefore, as a child, I can "give" experiential knowledge to myself as an adult, but I can't really teach the younger version of myself anything (unless I have a time machine). So, in a very literal sense, the child is the father of man because the child precedes his adult self.
In the "Ode," Wordsworth speaks of the loss of the fresh experiences of youth in addition to contemplations of one's own death. In my edition of the poem, Wordsworth quotes the last three lines of "My Heart Leaps Up" prior to the text of the "Ode."
The Child is Father of the Man;
And I could wish my days to be
Bound each to each by natural piety.
He wishes, as an adult, to be bound (loyal, dutiful) to that wondrous perspective with which he perceived the world as a child.
There is also a spiritual reference here. In certain Idealist philosophies and religions, the soul exists before a person is born and is something to which they return after death. Since the child is closer to the soul (in time) than the adult, the imprint of the soul on the child is more fresh. In other words, the child remembers more of the soul than the adult. In the fifth stanza of the "Ode," Wordsworth explains this:
Our birth is but a sleep and a forgetting:
The Soul that rises with us, our life's Star,
Hath had elsewhere its setting, (58-60)
The "Star" is a metaphor for the soul. That soul emerged in a different time and place than our birth ("had elsewhere its setting"). So, at birth we begin a process of forgetting that state of the soul. Wordsworth is speaking poetically and analogously here. Just as he wants to recollect his wondrous childish perception of the world, he sees a connection between this and his childhood recollection of that pre-birth soul.
Contemplating his immortality, the speaker looks to nature to give him a sense of the vivid and dreamlike experience of his youth, and to give him a sense of proximity to his soul. The child is the "father" because it is the childhood experiences which gave to the adult "intimations of immortality." These intimations may have come from that pre-birth soul, but they were manifested in new experiences in the world. Thus, a transcendent connection with nature, for the adult Wordsworth, is a connection to those intimations.