In chapter 11, we find some very specific descriptions of Dimmsdale's physical decline in the words:

While thus suffering under bodily disease, and gnawed and tourtured by some black trouble of the soul, and given over to the machinations of his deadliest enemy, the Reverend Mr. Dimmsdale had achieved a brilliant popularity in his sacred office. He won it, indeed, in great part by his sorrows.

I like how this description uses the term "black" and suggests his phsyical affliction is borne of his sorrow and guilt. This chapter is even entitled "The Interior of a Heart" as it is a study of his condition. It goes on to report that in private, Dimmsdale essentially tortured himself in an effort to purify himself from this great sin.

A quote I found for Chillingworth occurs in chapter 15, after the chapter titled "Hester and the Physician".

Hathorne narrates about him:

So Roger Chillingworth  - a deformed old figure, with a face that haunted men's memories longer than they liked - took leave of Hester.

Obviously in appearance, Chillingworth is a great fright to look at which would only arise out of a failing physical condition.