I think the veil has such a powerful effect on people because they do not understand it. When we don't understand things, we often fear them. Plus, when the minister first puts it on, he gives a sermon about "secret sins." During the sermon, each of the persons sitting in the pews thinks the minister is talking directly to him. So, on top of the fact that they do not understand the veil, they do know that veils are meant to hide things, the minister is talking about sins, OH NO! Maybe he knows what MY secret sins are, they fear. Remember that this is Puritan society and Hawthorne is criticizing the hypocrisy of going about acting "holier than thou" all the while sinning in secret. So the people are guilty. Their reactions to the veil are born from the guilt they have over their own sins. So a simple object like a veil is transformed by man's inner guilt and fears into a dark, forboding symbol.
The minister never takes the veil off. Even for weddings. This freaks the people out. They expect him to wear it when preaching, or at funerals, but at a wedding? Again, they don't understand it, so they fear it more. It is an ever-increasing puzzle. The more they cannot figure it out, the mored freaked-out they become.
At his death, he tells people:
I look around me, and, lo! on every visage a Black Veil
So even though the veil is worn by the minister, it might as well be on everyone else's face, "for all have sinned and fall short of the glory of God." The guilt is too great and the people transfer the symbol of the veil to themselves.
I guess they did not understand God's grace.