Yes, "Windsor Forest" is centrally about celebrating Britain and its empire.
In the first part of the poem, written when he was a teenager, Pope celebrates Windsor Forest as representative of England. As the poem opens, Pope compares Windsor Forest to no less than the Garden of Eden, stating that the "hills and vales" of the forest share Eden's beauty. He alludes to the celebrated English poet John Milton's Paradise Lost in describing the forest as "not Chaos like" but harmonious. He likens the forest to classical pastoral landscapes by imagining it populated with wood nymphs. All of this is meant to exalt England as on par with paradise, as well as the much-admired world of ancient Greece and Rome.
In the latter part of the poem, written several years later, Pope celebrates England's status as a rising world power. Britain signed the Treaty of Utrecht in 1713, which was favorable to British interests. The poem thus overflows with enthusiasm for the spread of British imperial power throughout the globe, stating:
The time shall come, when free as seas or wind
Unbounded Thames shall flow for all mankind,
Whole nations enter with each swelling tyde,
And seas but join the regions they divide;
Earth's distant ends our glory shall behold,
And the new world launch forth to seek the old.
In contradistinction to Spain, Pope envisions Britain bringing freedom, peace, and prosperity to countries like Peru and Mexico. This patriotic poem overflows with pride at a nation beginning to feel its centrality and power on the world stage.