The stranger is Roger Chillingworth, and it is described in the book that his slight deformity is "with the left shoulder a trifle higher than the right" (page 12 of the Holt Rinehart version). A deformity of any sort is meant to mark that character for some reason or other...here we understand that Hawthorne means to let us in on the secret that this man is Hester's husband and his cruel nature will become evident. First through his deformity and later through his treatment of Dimmesdale.
The man, we come to find out, is Roger Chillingworth, Hester's husband. It is said that he has the look of an intellectual about him--of someone who spent more time tending to intellectual pursuits than developing himself physically. The narrator states that the stanger is small in stature and has one shoulder that is lower than the other.