Ordinarily, you'd think that Mr. Hooper's parishioners would want to talk to him about his unusual habit of wearing a black veil all the time. After all, it makes him appear rather odd, and truth be told, it scares the living daylights out of many of them.

A deputation of Mr. Hooper's parishioners do actually try to confront him. When they go to see him, they're so embarrassed and uncomfortable at the whole situation that their courage deserts them. So Mr. Hooper continues wearing his black veil without anyone challenging him.

What this shows more than anything else is the level of deference that Puritan congregations traditionally paid towards their religious leaders. Pastors such as Mr. Hooper were important authority figures in the community, providing moral and spiritual leadership. Mr. Hooper himself is a well-respected figure in town. He's never previously given his parishioners reason to suspect that all might not be well with him.

It's understandable, therefore, that his parishioners are so reluctant to confront him about his wearing of the black veil. They don't believe that it's their place to inquire as to why Mr. Hooper has taken to behaving strangely. In this hierarchical Puritan society, people on the whole are reluctant to challenge authority figures. They are especially loathe to challenge those with spiritual authority like Mr. Hooper.