If you look at the last part of your quotation, the poet really does give you the answer.  Even the title, "tears, idle tears" provides you with part of the answer.  In the past which was sad but often fresh, he had time for tears, even for idle ones which means tears casually shed with no real fire to them.  "The days that are no more" is clearly telling you that those days are gone, that the fresh, sad, idle days of youth are gone when we have to leave that young time behind for the real world of adulthood.  Here, tears are for real, not used for an idle purpose, but to express true emotion in a real cause.