His physical deformity is a hunchback, which symbolizes much about his character. First, it is symbolic of his devotion to deep, as opposed to superficial, knowledge. He has studied apothecary and the healing arts, first in Europe and later among Native Americans. This is a sincere benevolent exercise until he discovers his wife's actions, whereupon he turns his skills toward the evil of revenge.
This reveals the other meaning of his deformity. His physical imperfection reflects his wounded soul. As the novel continues, his countenance appears more & more monstrous: an outward mirror of the twisted internal workings. Under this hideous visage he hides his true identity, so he can better torture the man who has cuckolded him. He entreats Hester
Breathe not, to any human soul, that thou didst ever call me husband! Here, on this wild outskirt of the earth, I shall pitch my tent; for, elsewhere a wanderer, and isolated from human interests, I find here a woman, a man, a child, amongst whom and myself there exist the closest ligaments. No matter whether of love or hate; no matter whether of right or wrong! Thou and thine, Hester Prynne, belong to me. My home is where thou art, and where he is. But betray me not!”
Thus, while he stands out due to his physical appearance, he is still anonymous in the land. His true identity is kept secret, while he is free to wreak his evil upon Dimmesdale.