The name of the house, "L'Abri," is significant because it means "shelter" in French. One would naturally think of a house as being a shelter, but for Desiree, the house turns out to be the opposite of a shelter when Armand allows her to leave.
Also, when Desiree's mother sees the house, she shudders, and the author describes it:
It was a sad looking place, which for many years had not known the gentle presence of a mistress, old Monsieur Aubigny having married and buried his wife in France, and she having loved her own land too well ever to leave it. The roof came down steep and black like a cowl, reaching out beyond the wide galleries that encircled the yellow stuccoed house. Big, solemn oaks grew close to it, and their thick-leaved, far-reaching branches shadowed it like a pall.
The diction of the description is much like a funeral, using words like "pall" and the color "black." As well, the house seems to be in disrepair; Madame Valmonde thinks that with a woman's touch, the house will be back to its former grandeur, but the reader discovers that it is not the shelter that Desiree needs. Its physical appearance reflects the mentality of its master, Armand.