The days of childhood seem to be so much longer than those of the speaker's present because for him, as for so many, childhood is a much more peaceful time and one that belongs solely to the child, whereas in adulthood much of the individual's time is not his/her own as obligations must be fulfilled during different parts of one's day.
And summer days, when we were young;
Sweet childish days, that were as long
As twenty days are now.
It is interesting that the speaker makes this reflection about time as he sits and contemplates the butterfly. Thus, there is evidence that "summer days" is an implied metaphor [the comparison of metaphor is not overtly made] for the carefree and innocent lives of people's childhoods. These were times in which a child could sit and meditate upon a butterfly and other marvels of Nature without interruption of any duty or responsibility. The speaker remembers childhood as a time in which he could enjoy the beauty and free spirit of the butterfly without interruption; in such a peaceful meditation, time in childhood virtually seemed to stop: "childish days, that were as long / As twenty days are now."