Chillingworth's obvious reason for "punishing" Dimmesdale is because the minister "stole" Hester from Chillingworth by committing adultery with her. In Puritan culture, this is an egregious crime, and the townspeople would have agreed that punishment for Dimmesdale was necessary if they had known that he had committed adultery.
Secondly, Chillingworth mentions to Hester that all the years that he has spent "torturing" Dimmesdale have changed him for the worse. He feels that he must punish the minister because had it not been for Dimmesdale's original sin, the "physician" would not have spent years turning himself into the repulsive being that he is when Hester meets with him. He says,
" 'And what am I now? . . . 'I have already told thee what I am! A fiend! Who made me so?' "
Chillingworth realizes and admits that he can never return to the calmer individual that he was when he first arrived in Boston; he has sacrificed that in his quest to punish the adulterer.
Chillingworth has a dual reason for hating and punishing Dimmesdale: he committed adultery with his wife, Hester, and he has the admiration and love of his community, something that Chillingworth could never have.
Adultery was considered a terrible crime in Puritan society. Even though Hester considered herself unmarried, as she thought her husband had abandoned her, she was still to live as a married woman.
Dimmesdale was his own worst enemy in that he couldn't admit to his congregation that he was guilty of sin. The people simply would not believe him. They admired and respected him for his holy lifestyle and his quality of sermon. They respected, revered and loved him in a way that Chillingworth, even if he were "local," could never attain.