The story is set on a plantation, ironically named l'Abri (which means the shelter) in Louisiana before the Civil War and the abolition of slavery. This location and these circumstances help us to see Armand Aubigny's terrible and, ultimately, tragic hypocrisy. He owns slaves, and his "rule was a strict one"—his slaves "had forgotten how to be gay" as they apparently had been when his father was alive and in charge. Furthermore, there are clues that Armand has sex with his slaves (an act which is more accurately called rape, but that is a different discussion): first, Desiree claims that Armand could hear their baby's cries "as far away as La Blanche's cabin." What possible reason would he have to be in a slave woman's cabin? Second, when Desiree tells Armand that her hands are "lighter than [his]," he responds by saying, "As white as La Blanche's." It sounds as though La Blanche is a particularly fair-skinned slave with whom Armand spends a great deal of time, and her skin is something with which he seems to have intimate familiarity.

Armand has no qualms about having sex with his slaves, individuals who he so clearly feels to be beneath him, and yet he is willing to outcast his wife for supposedly being partly black. He feels she has done "unconscious injury" to "his home and his name." Armand's treatment of his wife is appalling and would be regardless of when and where the story takes place; however, his racial hypocrisy, including his final act of destroying the evidence of his own mixed racial heritage, makes his actions even worse. In providing a way for us to observe his racial hypocrisy, the setting influences both the action and our interpretation of it.

The setting of "Desiree's Baby" contributes to the action of the story by showing the home of Armand and Desiree as falling into despair.

Madame Valmonde had not seen Desiree and the baby for four weeks. When she reached L'Abri she shuddered at the first sight of it, as she always did. It was a sad looking place, which for many years had not known the gentle presence of a mistress, old Monsieur Aubigny having married and buried his wife in France, and she having loved her own land too well ever to leave it. The roof came down steep and black like a cowl, reaching out beyond the wide galleries that encircled the yellow stuccoed house. Big, solemn oaks grew close to it, and their thick-leaved, far-reaching branches shadowed it like a pall. Young Aubigny's rule was a strict one, too, and under it his negroes had forgotten how to be gay, as they had been during the old master's easy-going and indulgent lifetime.

This allows an active reader to pick up on the foreshadowing of what is to come.  The home, "was a sad looking place" with "big solemn oaks".  One could come to understand that the setting echos the action of the story by enhancing the themes of sadness and solitude.

Therefore, knowing that each time Desiree's mother comes upon L'Abri she shudders, one can assume that the home and movement of the story will not be one filled with happiness and companionship.