Hawthorne is pretty scathing on the whole about the Puritans and their beliefs in The Scarlet Letter . Observe the preamble, where the narrator tells us how the book came to be written. His detailed description of the Custom House where he works and the surrounding wharf speaks of backwardness...
Though it's several centuries since the events of the story took place, it would appear that Salem is still stuck in the past, unable to adapt itself to the present day. To a large extent, this is because of the Puritan attitude that hangs over the town like a permanent thundercloud, enshrouding everyone and everything in a funereal pall of gloom.
The very visible signs of decay that disfigure the Custom House and its surroundings nearly symbolize the lack of cultural and intellectual progress in this part of the world. The world has moved on, but Salem hasn't. Although young women may no longer be publicly humiliated and paraded before the town for the sin of committing adultery, the spirit of Puritanism still lives on in the beams of the rotting wharf and the half-finished Custom House.
Customs are taxes paid on imported goods, and Salem, in its capacity as a port, deals with traders from all over the world. Yet because the governing Puritan ideology remains firmly stuck in the past, Salem is a port in seemingly terminal decline, a place that's seen better days.