Tom Walker and his equally miserly wife live in a house in New England that "stood alone and had a look of starvation." This "forlorn-looking" building is surrounded by a few scrawny, sterile trees and never has a fire lit in its hearth. It embodies the very isolated and unkind spirit that is possessed by its occupants, and, thus, is never visited by anyone. It is for this reason that "the house and its inmates had altogether a bad name."
Despite Tom's greed and his great desire for wealth, his anxiety after he sells his soul to Old Scratch in exchange for riches prevents him from enjoying his new status in life. His new house is described as "ostentatious" and he sets up a successful counting house in Boston, but his mind is tortured by fear that the devil will come any day to collect on his bargain. Indeed, this exact thing happens, and Tom is swept away by a black man on horseback who rides off into a thunderstorm. With Tom's soul stolen away, his possessions dissolve into cinders and wood chips, and his house burns to the ground. His greed ultimately leaves him with nothing.
At the beginning of the story, Tom's house is described as a "forlorn looking house that stood alone and had the look of starvation". Because of the stinginess and mistrust of both Tom and his wife, the house was as bad inside as it was out. Tom and his wife never trusted each other, they fought often, and neglected their animals. Their horse is pitifully described as having ribs sticking out, given barely enough food to survive. After Tom makes a deal with the devil, he builds a new, ostentatious home but never finishes it or furnishes it completely because he is too miserly. He still starves his horses and never takes care of his new possessions because that would cost money.