The author states specifically that the minister's black veil not only hides his face from the world but also gives him a darkened view of everything, including all the people he meets. The following sentence explains not only his subjective perceptions but his objective ones:
On a nearer view it seemed to consist of two folds of crape, which entirely concealed his features, except the mouth and chin, but probably did not intercept his sight, further than to give a darkened aspect to all living and inanimate things.
In other words, the whole world looks dark to him for the rest of his life. This seems to be an unusually morbid choice, since his view of the world and the people in it is probably melancholy enough without the veil. Hooper seems to be suffering from an excessive amount of guilt, and he also seems to be projecting that guilt out upon all the people in his community, especially those of his parish. He naturally makes everyone, including his own fiancee Elizabeth, feel uneasy. No doubt they realize they are not being open and candid with everyone all the time, but this is an impossibility, and no one really expects it.
Reverend Hooper seems to think that everyone in the village, male and female, should adopt his fashion and begin wearing black veils. This would be a weird sort of village, and it would be nearly impossible to get any work done. It wouldn't make anybody more righteous, only more secretive.
Seeing other people through a black veil apparently is effective in making Reverend Hooper think that everyone else is actually wearing a black veil which he or she is not aware of. It is an optical illusion. Furthermore, the effect his appearance has on other people looks to him as their natural state. It makes them reveal their secret guilts and secret fears, even their secret fantasies and secret crimes in their facial expressions and body language, because they believe he can see right through them in spite of the fact that they have been wearing protective masks, or personas, all their adult lives.