Hester Prynne notices changes in her then-thought-deceased husband, Roger Chillingworth ever since she first sees him in chapter 3, "The Recognition". Upon laying her eyes on her husband, the narrator (focalized through Hester's eyes) explains her reaction
...at the first instant of perceiving that thin visage, and the slight deformity of the figure, she pressed her infant to her bosom, with [a] convulsive  force..
Later on in chapter 8, at the Governor's hall, it has been three years since Chillingworth has been at the settlement. As Hester studies his decaying features, she correlates his smirk and his obvious malice as he stands next to Dimmesdale, to his looks.
Hester Prynne[...] was startled to perceive what a change had come over his features,—how much uglier they were,—how his dark complexion seemed to have grown duskier, and his figure more misshapen
Therefore, Hester interprets the changes to the way in that they connect to his personality; one which harbors more and more hatred as he comes along. He has turned darker, uglier, and misshapen. That is an allusion to the state of Chillingworth's soul.