The townspeople definitely do not like the black veil that hangs in front of Hooper's face. Some think it is a gimmick or a way to protect his eyes from the sun.
Others said there was no mystery at all, but only that the minister’s eyes were weak and needed to be shaded from the light.
But on the whole, the townspeople find it gives the minister a creepy and sinister vibe. Some of the people even get the feeling that the veil gives the minister some kind of supernatural ability to see each of their hidden sins.
Every listener, even the most innocent, felt that the preacher had crept up on them and discovered their hidden sins.
The best quote that shows the general feeling about the veil from the congregation comes very early in the story. Paragraph three specifically.
“I don’t like it,” muttered an old woman. “He has changed himself into something awful by hiding his face.”
The rest of the townspeople agree with that statement and feel that the mask has somehow turned their beloved minister into something dark. A great detail left by Hawthorne about the effects of the veil comes a few paragraphs later. He tells readers that the veil so upset some people during the church service that they had to leave. That is something that was practically unheard of during that time. Everybody went to church, and people simply did not walk out during the service.