There is another instrument of torture that should be mentioned as well:  Dimmesdale's "bloody scourge."  There is quite a large section of the text that deals with Dimmesdale's scourge.  Let's begin there:

His inward trouble drove him to practices, more in accordance with the old, corrupted faith of Rome, than with the better light of the church in which he had been born and bred. In Mr. Dimmesdale's secret closet, under lock and key, there was a bloody scourge. Oftentimes, this Protestant and Puritan divine had plied it on his own shoulders; laughing bitterly at himself the while, and smiting so much the more pitilessly, because of that bitter laugh. It was his custom, too, as it has been that of many other pious Puritans, to fast--not however, like them, in order to purify the body and render it the fitter medium of celestial illumination,--but rigorously, and until his knees trembled beneath him, as an act of penance. He kept vigils, likewise, night after night, sometimes in utter darkness; sometimes with a glimmering lamp; and sometimes, viewing his own face in a looking-glass, by the most powerful light which he could throw upon it. He thus typified the constant introspection wherewith he tortured, but could not purify, himself.

A scourge, of course, is a " whip or lash, especially for the infliction of punishment or torture."  So Dimmesdale's vigils are full of a lot more than prayer and fasting.  Dimmesdale is so bothered by his guilt that he actually whips himself in order to do penance for his sin with Hester.  Dimmesdale is often called a Christ-figure.  Dimmesdale's use of the scourge is one of the first implements that connect him to Christ (although the scourge is used to purify Dimmesdale for his own sins instead of the sins of humanity).  There are other connections as well, such as Dimmesdale's "three temptations" in the "desert" and his ultimate sacrifice.

Since there are no methods of atonement in the Puritan society, Dimmesdale must look for other ways to try and relieve himself of his terrible guilt.  Though Chillingworth tries time and time again to get Dimmesdale to confess his evident sins, Dimmesdale knows God is the only person he can confess to and until then, he must work out his issues here on earth. In Chapter XX "The Minister in a Maze" the narrator describes Dimmesdale in his study.  The quote shows the reader that, in an attempt for release, Dimmesdale had punished himself through vigils, prayer, and fasting.

Here he had studied and written; here, gone through fast and vigil, and come forth half alive; here, striven to pray; here, borne a hundred thousand agonies!

In Chapter XII "The Minister's Vigil" Dimmesdale returns to the spot where the novel began- the scaffold.  He climbs up and stands there hoping someone will come by and see him and question what he is doing.

There is also the question of why Dimmesdale continually places his hand over his heart. One day during an afternoon nap, Chillingworth looks at his patient's chest.  We know whatever he sees on Dimmesdale makes him very happy and seems to offer enough proof for whatever accusations he has concocted in his mind.

The physician advanced directly in front of his patient, laid his hand upon his bosom, and thrust aside the vestment, that, hitherto, had always covered it even from the professional eye. Then, indeed, Mr. Dimmesdale shuddered, and slightly stirred.

After a brief pause, the physician turned away. But with what a wild look of wonder, joy, and horror!