Dimmesdale pays a very high price. Everyday he feels the guilt of his sin as well as the guilt of having Hestor bear the burden of it. He suffers much for this. In addition, everyday he must face the respect of his congregation when he knows he doesn't deserve it. Perhaps worst of all, he must live without the woman he loves and see his daughter grow up not knowing he is her father. In many ways he feels he is not the man he would like to be to himself, his community, the woman he loves, and the daughter who doesn't know him.
Dimmesdale also, essentially, is killed because of it. He gets weaker and weaker, ultimately dying in Chapter 23 due to the fact that he has suffered so much.
What makes this self sacrifice worse is that this punishment is not limited to this life, but it will also affect them both in the afterlife as well. When Dimmesdale dies, he is sure to tell Hester that since they sinned together here, there should be no expectation that they would be able to be together later.