In much of this poem, the speaker addresses his beloved sister, Dorothy, whom he calls "my dearest Friend" in line 115.  As he addresses his sister, he is comparing his current reflective, more mature self with his younger self of five years earlier when he previously visited this beautiful wooded area.  Thus, it is easy to think that the word "thee" is used throughout the poem to also address Dorothy.

You are right, however,  that in the passage noted, lines 49-57, the speaker is addressing an element of nature.  In lines 55 and 56 he states:

How oft, in spirit, have I turned to thee,                        O sylvan Wye! thou wanderer through the woods...

The phrase "O sylvan Wye" is a direct appositive to "thee."  The Wye is a river that runs near Tintern Abbey; "sylvan" means "wooded."  So it is the river Wye, an element of nature, that is the "wanderer in the woods."  Look again at the entire passage; you can see that it is to this river that the speaker's spirts have often turned.  The river provides respite and peace for him in contrast to

...joyless daylight, when the fretful stir           Unprofitable (lines 52-53)...

As in so many of Willliam Wordsworth's poems, nature--in this case a river running through the woods--replenishes the human spirit.