Wordsworth is illustrating the ways in which the attitudes, feelings, and actions of adults are based on the attitudes, feelings, and actions of those same individuals as children. He says that he reacted to the sight of a rainbow in a certain way - his heart leapt at the beauty - "when my life began" and that, years later, he still reacts in the same way "now (when) I am a man." Furthermore, he expects that the same reaction will still be present "when I shall grow old."
Yes, that phrase is the theme of the poem. Childhood becomes the foundation for adulthood, preceding the later situations and shaping the patterns of later years in the same way that a father shapes and forms the actions taken by his child.