The answer to this question can be found if we look at what Mr. Hooper says on his deathbed at the end of his story and also when we consider the symbolic significance of the black veil. Note what Mr. Hooper says before dying, which relates explicitly to this issue:
Why do you tremble at me alone? ... Tremble also at each other! ... What, but the mystery which it obscurely typifies, has made this piece of crape so awful? When the friend shows his inmost heart to his friend; the lover to his best beloved; when man does not vainly shrink from the eye of his Creator, loathsomely treasuring up the secret of his sin; then deem me a monster, for the symbol beneath which I have lived and die! I look around me, and, lo! on ever visage a Black Veil!
What has led Hooper to separate himself from everybody else is something that he has become aware of but which others clearly are ignorant to. Hooper has realised that a metaphorical black veil of secret sin separates us all from each other. His realisation of this has led him to act on it, explicitly wearing the veil that implicitly we all wear on our faces. Hooper is the only character, however, who has the courage first of all to realise this and secondly to act upon it. His recognition of his secret sin, whatever it is, is something that will separate him from the rest of humanity, including Elizabeth, until his death.