The memory of snow is from many generations back. In fact, the memory is from generations and generations before this current Receiver (who we know as The Giver) was alive, and prior to the Receiver before him. Because the memory is so old, it takes a lot more energy to pull forward and then to transmit.
"It's a very distant memory. That's why it was so exhausting - I had to tug it forward from many generations back" (CH.11).
If you relate it to yourself, it would be a bit like trying to remember a dream you had when you were three or four years old. It would be an old memory that would take a lot of work and energy to recall, if you even if could.
What this tells us about the community is that they have not had snow, or hillsides, or color, for a very long time. They moved to Sameness generations ago and that this memory is from such a distant past also indicates how accustomed the community members are to this "new" way of living. Although for readers this would sound like a "new" way of living, in the book this society has been doing this for probably hundreds of years. Later when Jonas rebels against some of these ideas, we better understand how hard it will be for the community to go back to the ways of life that existed pro to Sameness.
The Giver remembers things not just from his own lifetime, but from those of previous Givers and their communities. The further back in time a memory (that is, the more Givers it is remembered through), the more difficult it is to transmit due to time and accumulation of memories. One thing this reveals is that it has been a very long time since the "perfect" society of Jonas' time has experienced snow--or went to sameness.
In addition to what lynn30k has written, you should consider the complexity of this particular memory. There is so much involved! Think of the difference between a stick figure and a marble sculpture: the effort required to convey a complex memory is of course much more than the effort required in conveying a simple one.