In a lot of Romantic poetry and in many of Robert Frost's poems, the poet/speaker is very much in touch with nature and uses nature's imagery and analogous connections to human life (i.e., cycle of seasons, cycle of life) to express a transcendent experience. In "Tree at My Window," the speaker does the same in comparing himself to the tree but says that his inner thoughts are more "profound" than any literal or metaphorical significance he can get from the tree:
Not all your light tongues talking aloud
Could be profound.
In the third stanza, the speaker mentions that he has seen the tree "taken and tossed" just as the tree (personified) has seen him "taken and swept / And all but lost" in dreams. The speaker looks at the tree like a friend in that they both have similar or analogous functions. The tree deals with outer or external storms and all manner of good and bad weather. The speaker deals with metaphorical storms and all manner of good and bad emotions and experiences.
The speaker appreciates that fate (personified) has brought he and the tree together. Fate/nature has made this analogy possible. As the tree experiences periodic moments of calm, storms, warmth, cold, the speaker experiences periodic moments of happiness, grief, love, and despair. The speaker realizes his thoughts are more "profound" than the tree's external engagements with the weather, but the speaker also appreciates that this odd brotherhood between he and the tree exists. This is why he says:
But let there never be curtain drawn
Between you and me.