The theme is one of (unwanted and regrettably) inevitable change over time toward old age. Yet another theme runs parallel to that of the changes of time and that theme is concerned with the speaker's fullness of experience.
The speaker laments the fact that time has made him old, but he defiantly proclaims his energetic relationship to life's experiences. Just because he is now older than he once was does not remove him from life or disqualify his substantial and important encounters, episodes, and appreciations of the world.
He once was right in the thick of life, as it were, and he is proud of this fact. He holds tightly to it.
My chair was nearest to the fire
In every company
That talked of love or politics
The speaker staunchly defends the validity of his experiences and his continuing possession of the consciousness of those experiences. Furthermore, he still appreciates beauty when he sees it, though he compares or connects this beauty to his former experiences.
There's not a woman turns her face
Upon a broken tree,
And yet the beauties that I loved
Are in my memory
Time may have forced him to take "shelter from the rain/under a broken tree" but his experiences are intact, his lust for life is intact and his appreciation of beauty is intact. He resists aging and the changes of time -- or is compelled to "spit in the face of Time" -- because he is not yet done with life. He has a stake in life that is not to be undone and not to be erased.
Age may make him into a broken tree, but the fact that he grew once in full flower is perhaps more important than anything else. The story of that tree and of the man are stories of becoming, of growth, of life lived. Thus time is the mechanism of the man's undoing via age but also the mechanism of his living, his glory, his joy, etc.