You are right in pointing towards the massive transformation that occurs in Mr. Hooper after he dons his black veil, both in terms of his appearance and what others make of him. However, to consider how he was viewed before he decides to go through this change, you need to look towards the beginning of the story, where the narrator clearly outlines his standing in the community where he ministers:
Mr. Hooper had the reputation of a good preacher, but not an energetic one: He strove to win his people heavenward, by mild persuasive influences, rather than to drive them thither, by the thunders of the Word.
Thus we can see that although Mr. Hooper had a good reputation, he was not famed for the power and vivacity of his preaching, but rather known for his mildness and gentleness. Of course, the black veil changes radically his position in the community and the effectiveness of his job as a minister, as people come to identify their own fallen nature in his black veil.