The term “retreat” can have more than one meaning. In a military sense, a retreat is when you withdraw in the face of a superior enemy. (You run away, or pull back.) In daily life, a retreat is also pulling back, but more in order to get out of the stresses of daily life and reflect on deeper matters. Many religious organizations have period retreats, often with specific themes.
This poem is a rather wistful meditation on both possible meanings but mostly the second. Vaughn is moving away from the present back to the past, pulling back from the sinful, earthly focus he has developed in recent years, in the wish of reclaiming his original innocence.
However the depth of the poem is in its impossibility. He’s saying, essentially, I wish I could go back to when I was pure…but I can’t. The knowledge he’s gained of this world has changed him.