At the first service, the parishioners are unnerved by the Minister's veil and most feel a sense of dread, something foreboding. The narrator notes that Mr. Hooper's sermons had always been mildly persuasive; never loud and instructing them with the "thunders of the Word." During this sermon, he is still the same calm orator, but the effect on his listeners (auditors) is different.
. . . there was something, either in the sentiment of the discourse itself, or in the imagination of the auditors, which made it greatly the most powerful effort that they had ever heard from their pastor's lips.
The subject had reference to secret sin, and those sad mysteries which we hide from our nearest and dearest . . .
The listeners are affected by this sermon (because of the subject, the foreboding veil, or both), but their assumptions about the veil have to do with Mr. Hooper. They don't consider that the veil is a symbol of their own secret sins as well. (This is the message Mr. Hooper tries to impart at the end of the story.)
At the funeral, the veil seemed appropriate for this solemn occasion. However, it still leads to mysterious speculation. Upon leaning over the coffin, his veil drooped and he pressed it back to his face. One person thought the corpse shuddered when Mr. Hooper's face was exposed to the corpse. "I had a fancy," replied she, "that the minister and the maiden's spirit were walking hand in hand." One implication is that this person was thinking Mr. Hooper was hiding a secret only he and the maiden (the deceased) knew about, perhaps an affair.
At the wedding, the veil seemed inappropriate. And the guests felt that it brought "nothing but evil to the wedding." The bride was so horrified by the veil that she grew pale enough to cause a whisper that she was the maiden who had died earlier, now returning to be married. At the end of the wedding service, Mr. Hooper catches his own reflection and is also horrified. He rushes off.
In both cases, the veil causes feelings of dread but also solemnity. These are the effects of the veil. And though Mr. Hooper's veil may indicate his own secret sin, through his veil, he sees a veil on everyone's face, indicating they all have secret sins as well. Ironically, he exposes his secret sins with the veil; they all hide theirs without a symbol such as the veil. In other words, their sins are veiled by not wearing the veil.