A good place to start in answering this question would be Chapter Ten, entitled "The Leech and his Patient." This gives us a real insight into both of these two characters and the way that their appearance is shaped by what is going on within them.
Note how Chillingworth's eyes are referred to in a frightening, almost supernatural way as he engages in his work as a "miner":
Sometimes, a light glimmered out of the physician's eyes, burning blue and ominous, like the reflection of a furnace, or, let us say, like one of those gleams of ghastly fire that darted from Bunyan's awful doorway in the hillside and quivered on the pilgrim's face. The soil where this dark miner was working had perchance shown indications that encouraged him.
Clearly this description of the "blue and ominous" light that is compared to the "ghastly fire" indicates how Chillingworth is depicted as an almost demonic or evil character in his determination to find out Dimmesdale's secret.
When we consider Arthur Dimmesdale, you might want to think of the final paragraph in Chapter Nine in terms of how the description we are given of him reflects his inner anguish:
Alas! to judge from the gloom and terror in the depths of the poor minister's eyes, the battle was a sore one, and the victory anything but secure.
Clearly, the inner guilt that dominates Dimmesdale's "spiritual sickness" is impacting his physical appearance and making him appear sick, pale and distraught.
Thus both characters' physical descriptions are shaped by what is going on with them internally. On the one hand, Chillingworth's somewhat demonic appearance is justified by his obsession with "mining" the secrets out of the interior of Dimmesdale's heart. On the other hand, Dimmesdale is clearly shaped by his own inner guilt that is eating him up from the inside.