The setting of "The Minister's Black Veil" is the town of Milford, somewhere in New England, which we know from a footnote in which Hawthorne says that "another clergyman of New England" made himself famous for adopting the same "eccentricity" as Reverend Hooper. Given the fact that Hawthorne spend much of his life in Salem, Massachusetts, it is reasonable to place Milford in or near Massachusetts.
Although the fact that the story takes place in "Milford" is of little consequence, the fact that the setting is in New England is very important. In towns like Milford, usually small and agricultural based, the minister is usually the leading citizen, looked up to by everyone, including civil authorities, for moral guidance, religious instruction, and comfort in times of distress. Anything that alters the nature of the town's minister will also affect the town either negatively or positively, so the setting itself is an important element in the overall story even though the town's name is relatively unimportant.