The adjective "downy" does seem to suggest that the snowflakes are being compared to "down," which is defined as a noun meaning the soft, first plumage of many young birds or the soft under-plumage of birds. The word "downy" in this sense would mean "down-like." Goose down is commonly used for stuffing pillows and comforters. Eiderdown, the soft feathers from the breast of the female eider duck, is especially prized for such purposes. The ducks pull the feathers out of their breasts to make nests in breeding season, and humans will gather these nests to sell for very good prices.
The word "downy" also might have been intended to suggest the idea of "falling," or moving downwards, which can be felt throughout the poem, especially in its seemingly downward-trending rhyme scheme. The AABA rhyme scheme in the first stanza becomes BBCB in the next. The B in the first stanza is picked up and continues falling throughout the second stanza, and so on. But the main comparison seems to be between softly falling, extremely light snowflakes and the white down of birds.
If we wanted to stretch this comparison further, we might say that the word "downy" suggests that the speaker, sitting there in the cold, is thinking of his own warm bed with a down-filled comforter and a down-filled pillow.