The Giver's former spouse lives with the Childless Adults.
The Giver is quite old, and in the community, it is customary that "when adults...be(come) older, their lives be(come) different. Thay (are) no longer needed to create family units. Jonas's own parents, when he and Lily (are) grown, (will) go to live with the Childless Adults", as has the Giver's former spouse.
The Giver and his spouse did actually have a child at one time. She was the Receiver-in-Training ten years before Jonas, and her name was Rosemary. Rosemary was unable to accept the pain of the memories she had to bear, and requested release. After Rosemary's request was granted, assuming that there was no other child in their family unit, there was no further need for the Giver to stay together with his spouse. At that point, his spouse would have been sent to live with the Childless Adults.
Jonas will have the opportunity to take a spouse if he chooses, but as the future Receiver of Memories, his life with her will be very difficult. The books that Jonas will be allowed to access will be forbidden to his spouse, and he will not be able to discuss any part of his work as Receiver with her. It was hard for the Giver to adhere to these rules when he had a family unit, and it will be difficult for Jonas also. The memories he bears will be the center of his life, but he will not be able to share them with his partner (Chapter 13).