There are a few reasons for the Giver to feel this way. First, there is the sheer quantity and age of memories he must hold. Second, it is a burden to hold memories and never be able to share them. And third, he is responsible to the community if any memory is needed. This is a terrible burden for him.
The Giver is essentially responsible for remembering everything that occurred before the community went to Sameness. It is exhausting to the Giver. For each memory, he must "tug it forward from many generations back" (83). We get the sense that this is almost a physical drain upon him, the older the memory, the more work needed to do this. And there are so many memories. Even sharing one with Jonas makes him feel lighter, "A little weight off this old body" (82).
When Jonas and the Giver make the decision that Jonas is to leave the community, they talk about the memories that the Giver has and that Jonas now has, love and pain in particular. The Giver tells Jonas that memories are meant to be shared, saying, "The worst part of holding the memories is not the pain. It's the loneliness of it" (154). Imagine how lonely it would be to never be able to share your memories with another. Much of our joy in memory lies in the sharing of it.
It is also a burden upon the Giver because he is the sole repository of all and any memories the community might need if it encounters a problem that is from Elsewhere. There is no one for the community to turn to except the Giver for advice on a problem from outside the community. This is a grave responsibility for one person to have.
While being the Giver has been an honor, it has also been a great burden. It's a lonely task, with great responsibility and hard work.