As you note in your question, aside from Reverend Hooper, who, after donning the black veil, spends the rest of his life isolated from his congregation and the larger society and, one can argue, isolated from himself.
The only other character who suffers a similar isolation is Hooper's fiancee, Elizabeth. Early in the story, she tries mightily to convince Hooper to remove the veil, which he refuses to do, and she even asks him to "[l]ift the veil but once, and look me in the face." This is her final, desperate attempt to re-establish some normalcy in Hooper's behavior; unfortunately, Hooper does not lift the veil, and Elizabeth leaves him (forever, we assume).
At the climax of the story, as Hooper is on his deathbed, still refusing to lift the veil, we learn that he is being attended by "the nurse, no hired handmaiden of death, but one whose calm affection had endured long, in secrecy, in solitude. . . ." Hawthorne tells us that Elizabeth's "calm affection," her love for Hooper, had lasted all the intervening years. We learn here that another person, one whose life we had no knowledge of after she left Hooper, had continued to love him in silence and isolation for decades.
Elizabeth's solitary and lonely life is a sad, unintended consequence of Hooper's wearing the veil. From that perspective, many readers criticize Hooper's behavior as prideful and selfish--it is one thing to ruin one's own life, it is another thing altogether to ruin the life of another.