The "old pensioner" has been changed by the passage of time. He may no longer be actively involved in "every company that talked of love or politics"; he no longer joins in the fights with weapons or words against conspiracies or "human tyranny." Passing women no longer even look at him as they go by.
In spite of all of this, perhaps because of all this, the old man hangs on to his memories. He is not conceding to time, not giving up on life and his ability to take whatever part he still can in it. He still has his mind and the capacity to understand how "Time...has transfigured me."
Understanding does not mean acceptance, however. The pensioner is fighting the changes overtaking him. "I spit into the face of Time that has transfigured me." The theme of the poem is the pensioner's determination to resist, to retain the "beauties" and memories of his life and his activities, the lessons he learned before he was "transfigured" by Time.
The theme is one of (unwanted and regrettably) inevitable change over time toward old age. Yet another theme runs parallel to that of the changes of time and that theme is concerned with the speaker's fullness of experience.
The speaker laments the fact that time has made him old, but he defiantly proclaims his energetic relationship to life's experiences. Just because he is now older than he once was does not remove him from life or disqualify his substantial and important encounters, episodes, and appreciations of the world.
He once was right in the thick of life, as it were, and he is proud of this fact. He holds tightly to it.
My chair was nearest to the fire
In every company
That talked of love or politics
The speaker staunchly defends the validity of his experiences and his continuing possession of the consciousness of those experiences. Furthermore, he still appreciates beauty when he sees it, though he compares or connects this beauty to his former experiences.
There's not a woman turns her face
Upon a broken tree,
And yet the beauties that I loved
Are in my memory
Time may have forced him to take "shelter from the rain/under a broken tree" but his experiences are intact, his lust for life is intact and his appreciation of beauty is intact. He resists aging and the changes of time -- or is compelled to "spit in the face of Time" -- because he is not yet done with life. He has a stake in life that is not to be undone and not to be erased.
Age may make him into a broken tree, but the fact that he grew once in full flower is perhaps more important than anything else. The story of that tree and of the man are stories of becoming, of growth, of life lived. Thus time is the mechanism of the man's undoing via age but also the mechanism of his living, his glory, his joy, etc.