Most of The Memory Keeper's Daughter takes place in Lexington, Kentucky, a medium-sized city but small enough for neighbors to acknowledge one another at the supermarket and also to keep track of one another's business. This explains why Caroline Gill must leave town with Phoebe, the young baby that David Henry rejects. For this reason, much of Caroline's story is told in another setting, in Pittsburgh, Pennsylvania.
There is no real significance to selection of Kentucky as a setting. The story could take place in any small town. Much of the action occurs within the two Henry houses: the house they bought when first married and the second house they buy to try to get away from the memories of their so-called dead child. The significance of the two houses is symbolic. The first one holds their fantasies of what their lives would be like as they begin to create a family. The second one holds their attempts to re-create those dreams. Throughout most of the story, the Henrys own both of the houses simultaneously. They are unable to completely give up the old one for the new. So the first home lies vacant—as empty as their marriage.
In contrast to the green open spaces and landscaped urban lawns of Lexington is the place where Caroline lives. At first, Caroline shares a home with her new-found friend, Doro. The house has been in Doro's family for a while and Pittsburgh has grown up around it. Once idyllic, the house now sits close to a freeway. Pittsburgh is an industrial town, noisy and often blackened with soot, at least in the 1960s. But even though David and Norah have the healthier environment, their psychological worlds are terribly polluted. It is Caroline who has a clear conscience.
As for the twins, Paul's setting provides every luxury he could want, but it is Phoebe's setting that provides what she needs. In the setting where Paul lives, he struggles to find love. In the setting where Phoebe lives, she exemplifies love.
For a brief time, first through David's memories and then through a short trip that he takes, the story is set in West Virginia. It is here, in a delapidated, small house that David lived as a child. It is to this abandoned house that David returns when he fully acknowledges the consequences of his having given away his daughter. In an attempt to redeem himself, David offers a home to the young teenage girl who has been squatting on the family property. Rosemary is pregnant, and David feels compelled to save this unborn child.