Both Chillingworth and Dimmesdale let emotions like guilt or revenge take away their joy in life.
Dimmesdale spent much of his time doing secret penance for his sleeping with Hester and fathering her child. While he begged her to reveal him as the father, he himself could not make himself come forward with the truth. His inability to confront his fear led to his body wasting away under the burden of guilt. The quality of his life would have been better if he had confronted the fear and faced the public disdain. Hester had survived, and even earned respect over the years after her public humiliation. Dimmesdale is to be pitied for suffering silently, when the truth would have led to a more peaceful life.
Chillingworth allowed a single-minded pursuit of revenge to take away any resemblance of humanity to his person. By refusing to let it go, he transformed into a dark and malevolent creature. His life was destroyed b y his rage and need for revenge.
Both men destroyed their lives by allowing their guilt and revenge to overtake their lives.
It is difficult for anyone to "pity" these two characters as both men made their own decisions and created their own problems. Both men were deceitful and only Hester owned up to her actions and was honest. She might be the only one we could legitimately pity as an individual character.
However, what we can really pity is the situation and the strictness of Puritan values that made people suffer for being human. Hester was made an outcast by one mistake and she was punished beyond what would seem reasonable.
Hester's daughter is the one who deserves the most pity because this child is made to be an outcast and seen as almost demonic because of society and not because of anything she herself did.