"Setting" comprises not just a geographical location, but also the historical/ social context and the time period when the story takes place.
The geographical setting of "The Minister's Black Veil" is in New England, particularly in a small town named Milford. Hawthorne traditionally sets his stories (and novels) in actual locations, some of which come with a backstory as in the case of "The Custom House" in The Scarlet Letter, for example.
There is, indeed, a "city of Milford" in Massachusetts, but we can only assume that Hawthorne used this as yet another real location on which to base his story. This is because it is not certain whether Hawthorne was referring to "the" Milford, Massachusetts or if he just happened to use that name for no real reason.
An important dimension of the setting is the time period in which the story is set, which is colonial New England. By "colonial", we refer back to the establishment of the original colonies, preferably parting from the passing of the First Navigation Acts in 1651, and all the way until the American Revolution.
The colonial period in New England, where Milford is located, is more significant in 1686. This is the year when the actual "Dominion of New England" is created by King James II through the unification (combination) of 8 colonies into one, for several reasons, none of them punitive. The colonies included Rhode Island, Connecticut, Massachusetts, and Plymouth. In 1688, East and West Jersey (New Jersey) and New York were included in the dominion.
Of further importance is the social setting of the story. Colonial New England was also puritanical. The term "puritan" is now used loosely but, at one point, it was a pejorative used against Anglicans from the Church of England that thought that the church needed to clean up its act and be, well, "more pure".
Therefore, those incoming Anglican colonists who settled in Plymouth were separatists who believed that the Church was corrupted and believed that those who were truly Christian should get away from it.
The colonists from Massachusetts Bay, however, while also felt that the church needed some serious cleaning up, did not want to separate from it. Imagine how crazy things got when everyone, separatists and non separatists, had to become one under one same dominion.
The set up of the story, where the parson seems to be at the epicenter of the dynamics of the villagers (or acts as if so), is typical of the time period. The figure of the minister is quite influential, and what the parson did must have indeed caused a commotion. After all, the story itself is based on real life events with "another parson from New England" named Joseph Moody from York, Maine.
Therefore, Hawthorne kept quite true to the time period he wished to illustrate.