In chapter 13, The Giver explains that he used to have a spouse when he was younger, and he cautions Jonas against applying for one because it is difficult to have a spouse when you cannot share any of your work. He also states that since books are forbidden to ordinary citizens, the living arrangement would be different. Since the spouse could not be allowed to see the books, the Receiver of Memory has to hide them.
The society in The Giver is one of very tight control. Spouses and children do not have the same relationships as in our world. They are assigned by a committee. You don’t choose your spouse, and you are not biologically related to your child. In the case of the Receiver of Memory, there are so many secrets that need to be kept from the spouse that the conditions would be even more difficult. It is a very lonely life.