When Jonas decides to leave, he pleads for The Giver to come with him. The Giver responds that he is old and not strong enough for the journey. He also says that the community will need him.
After Jonas leaves, they know he will face harsh conditions. Long hours of running, harsh weather and lack of food would be very hard on The Giver. He would likely slow them down. Gabe is enough of a burden, since he too needs to eat and be kept warm and cannot fend for himself. The Giver has also given Jonas many of his memories, possibly leaving him not enough to sustain himself and them.
The main reason The Giver cannot leave is that when Jonas leaves, all of the memories he has will be released back to the community. Chaos will ensue unless someone is there to help them. With both Jonas and The Giver gone, all of the memories of pain and suffering would go back to the people. They would be confused and not know what was happening to them, and without The Giver there would be no one to help them through it.