Your question is interesting because we are never told explicity the reason why Mr. Hooper dons the black veil, except for it being a symbol. Other authors, in particular Poe, have suggested that there was some kind of illicit relationship going on, but there is no proof other than mere conjecture. When we think about the conflicts in this story, what is fascinating to me is how Hawthorne used the story to challenge Puritanism and the conventions of his time - it is this central moral conflict that is the driving force of the story.
"The Minister's Black Veil" exposes the hypocrisy and judgemental attitude of many Puritans. It does this through pointing out that everyone has some form of secret sin that we hide, perhaps even from ourselves. Consider Mr. Hooper's final declaration regarding his black veil:
"When the friend shown his inmost hearth to his friend; the lover to his best beloved; when man does not vainly shrink from the eye of his Creator, loathsomely treasuring up the secret of his sin; then deem me a monster, for the symbol beneath which I have lived, and die!"
Hawthorne thus makes a serious allegation about the form of religiosity that Puritans presented - in outward aspect true and good, whilst on the inside sins and faults were harboured and not exposed. It is man's inability to completely be honest and open about his failings and sins that make the black veil so terrifying an image in the story, for everyone, at least partially, acknowledges that they have a black veil guarding their faults just as surely as Mr. Hooper does.
Thus this story seems to talk more about hypocrisy and judgemental attitudes which are used often to hide and mask our own secret sins. This is the moral conflict of the story that demands that we all recognise our own black veils.