In "The Minister's Black Veil: A Parable," why do the parishioners have such an intense response to seeing the minister's veil?

Mr. Hooper's parishioners have an intense reaction to his black veil, at first, because it is extremely off-putting.  They are very confused by it because it is so uncommon a thing for a person to cover up their face in this way; we especially like to see the eyes of any person to whom we speak.  They first wonder if it is even their minister behind the veil, and then one old woman says that "'He has changed himself into something awful, only by hiding his face," and another man cries, "'Our parson has gone mad!'"  It is such a strange thing for them to see that they question his motives or even if he has simply lost his mind.

Later, though, especially after he gives his first sermon while wearing the veil, Mr. Hooper's congregation starts to feel very nervous and vulnerable.  The subject of this sermon

had reference to secret sin, and those sad mysteries which we hide from our nearest and dearest, and would fain conceal from our own consciousness [....].  Each member of the congregation [...] felt as if the preacher had crept upon them, behind his awful veil, and discovered their hoarded iniquity of deed or thought.

Mr. Hooper essentially admits that he has "secret sin" himself, sin that he would prefer to hide from everyone, even his loved ones, and he insinuates that his listeners each do too.  They hear him speak and they "quaked" because they felt, suddenly, that he knew that they had secrets too.  For this reason, their behavior toward him changes, and even his fiancee leaves him once she realizes what the veil symbolizes.