On a really basic level, one reason the townspeople react so strongly to the black veil Mr. Hooper wears is that it is odd. It is a strange thing to do to cover one's face as he has done, and it is especially off-putting that they are unable to see his eyes when they know that he can see them. Even before he speaks on the day he first dons the veil, his parishioners doubt his identity, his intentions, and his sanity.
Further, the first sermon he delivers after putting the veil on addresses "secret sin, and those sad mysteries which we hide from our nearest and dearest, and would fain conceal from our own consciousness, even forgetting that the Omniscient can detect them." This sermon makes the entire congregation quake in their seats, feeling as though the minister has discovered their own secret sins. In fact, he has. He wears the veil as a physical symbol of the way each person hides their secret sinfulness from the world, themselves, their loved ones, and even God; everyone wears a veil, he implies, it is simply an invisible one.
Another reason that the townspeople seem to fear the veil is that they do not completely understand it. While they seem to possess some shadowy knowledge of its meaning, as we see from their response to Mr. Hooper's sermon, they don't fully understand and are afraid to ask outright. They send a contingent of church leaders to speak with Mr. Hooper on the subject, and even these cannot bring themselves to inquire specifically about the veil. We tend to fear what we don't understand, but even those who do seem to get it -- like Mr. Hooper's fiancee, Elizabeth -- fear it because to acknowledge its meaning is to acknowledge its truth: if one admits to understanding what the veil represents, the way each of us hides our true, sinful natures from everyone else, one admits that what it represents is real. One essentially admits to having secret sin, and this is what, according to Mr. Hooper's first veiled sermon, we most fear to do.