The speaker of the poem is a tramp who likes to spend his whole life wandering from place to place, out in the open air come rain, hail, or shine. He wants us to know that the change of seasons doesn't bother him at all. Spring, summer, fall, winter—whatever the season, whatever the weather, you'll always find him out and about wandering.
Come the winter time, the wanderer will still be roaming far afield. Although the winter may have silenced the birds in the trees, forcing them to migrate to warmer climes, the wanderer has no intention of going anywhere. He'll stay right where he is, even if the bitter cold of winter should bite into his blue, frosty fingers. If you've ever been outside on a cold winter's day without gloves, you'll know that your fingers can very quickly turn blue in the icy chill, and that's most probably what the speaker's referring to here.
The wanderer wants us to know that the prospect of spending all winter outside in the freezing weather doesn't concern him in the slightest, blue fingers or not.