There is a wry, dark humor in the narrator's description of the Salem Custom House. Hawthorne himself worked there and didn't think much of it. It was a place where people, usually merchants, would pay customs or taxes on goods they were importing for resale from overseas.
The narrator describes the Custom House as rundown place, dingy and full of cobwebs. He takes humorous jabs through the Custom House at Salem itself as a faded and no-longer-important port. Further, his elderly fellow employees are not prone to work very hard or be very competent, as most got their jobs through connections and not any particular qualifications for the work. The narrator describes them drily as:
a row of venerable figures, sitting in old-fashioned chairs, which were tipped on their hind legs back against the wall. Oftentimes they were asleep, but occasionally might be heard talking together, in voices between speech and a snore, and with that lack of energy that distinguishes the occupants of almshouses, and all other human beings who depend for subsistence on charity ...
Hawthorne is being ironic, or meaning the opposite of what he says, when he calls the men who are asleep or half asleep in their chairs "venerable," which means worthy of honor or respect. He is also humorous in a barbed way when he likens their jobs to a form of charity.
It is, however, while the bored narrator is working at the custom house that he finds the scarlet letter and the account of Hester Prynne. However, it is not until he is laid off from the stultifying job that he can muster the creative energy to tell Hester's story as he imagines it to have happened.
The dull custom house thus provides a frame and foil for a much more compelling and romantic tale from a former time.