When the speaker of this poem says, "my manhood is cast / Down in the flood of remembrance" in the poem's final lines, he compares the rush of memories he experiences to a flood of water using a metaphor. A metaphor is a comparison of two unlike things where the poet states that one thing is another. As the speaker listens to the woman singing and playing the piano for him, it takes him back to his childhood, and the metaphor comparing his memories to a flood implies that the memories come in such a great quantity and at such a fast speed that he is overwhelmed by them (and this would be why he "weep[s] like a child for the past" in the final line of the poem).
Further, when he says that his "manhood is cast / Down," he is taking something intangible, his "manhood," and suggesting that it is subject to physical forces, being "cast / Down," and it can be thrown away into the flood of memories, like an article of clothing or an unwanted object; this is another metaphor (though not quite as clear as the memories=flood metaphor) because many different things could be "cast / Down." What is certain, however, is that one cannot literally cast down one's manhood, and so we know the comparison is figurative.