It is clear from this short story that Mr. Hooper has not chosen to wear this veil for any frivolous reason or for a joke. His decision to start wearing the veil is linked to his identity as a Christian and as a man before God, and has been taken with utmost seriousness and reflection. Note the way that Mr. Hooper himself explains why he has done what he has done to Elizabeth when she asks for a reason for his somewhat strange behaviour:
"Elizabeth, I will," said he, "so far as my vow may suffer me. Know, then, this veil is a type and a symbol, and I am bound to wear it ever, both in light and darkness, in solitude and before the gaze of multitudes, and as with strangers, so with my familiar friends. No mortal eye will see it withdrawn. This dismal shade must separate me from the world: even you, Elizabeth, can never come behind it!"
Mr. Hooper describes his decision to wear the veil as a "vow," which indicates the seriousness of that decision. The symbolic function of the veil, which acts as a reminder of the sin that all humans bear and are unable to ignore, is something that applies in every relationship, not just relationships where one is distant or doesn't know others very well. Therefore Mr. Hooper is right to refuse to take off the veil, even at the cost of losing Elizabeth. He could not wear the veil for some and not for others and have the veil continue to represent the same meaning.