There are a variety of physical locations that are described in "The Devil and Tom Walker", and the entire story does not take place in any one of them (unless we consider the entire Massachusetts country side to be representative of moral decay...probably not a good thesis). However, there are two particularly significant physical settings that are described in detail, and which suggest moral decay; the Walker homestead, and the swamp.
The homestead is described as a fairly miserable place, not only in its appearance, but in its reputation among the neighbors and its general tone ("The house and its inmates had altogether a bad name."), due to the constant arguments and fighting between Tom and his wife. Physically, the house is described as forlorn, surrounded by sickly trees ("emblems of sterility"), surrounded by unproductive fields, and likened to being a prison. This moral decay is largely limited to the Walkers themselves; if we consider morals to be a matter of "doing what is right", then the Walkers literally live in wrong-ness; there is no moral gray area here, and their book truly can be judged by its cover.
The swamp represents a "descent into darkness" in Tom's character, but is also emblematic of social decay, moreso than the Walker home. As Tom enters the swamp, it is described as;
-an ill-chosen route. The swamp was thickly grown with great, gloomy pines and hemlocks, some of them ninety feet high, which made it dark at noonday and a retreat for all the owls of the neighborhood. It was full of pits and quagmires, partly covered with weeds and mosses, where the green surface often betrayed the traveller into a gulf of black, smothering mud...
and so forth, complete with lurking alligators. The swamp is every stereotype ever conceived of about a swamp, and Tom's passage through it (particularly his choice to remain on this path) is representative of his moral decay. The aspect which pertains to social decay is largely found in the trees which the devil is chopping;
Tom looked in the direction that the stranger pointed, and beheld one of the great trees, fair and flourishing without, but rotten at the core, and saw that it had been nearly hewn through, so that the first high wind was likely to blow it down. On the bark of the tree was scored the name of Deacon Peabody-
Irving is being so straightforward with the symbolism here that he's practically hitting us over the head with it; the Deacon's moral decay is represented by physical decay in the form of rot. To top it off, the devil declares, of the tree representing the Deacon, "He's just ready for burning!", suggesting that the Deacon's fate is not a pleasant one.