Dimmesdale would often grab his chest, especially when particularly difficult or painful experiences come up. Here's an example from Chapter 10:
"Then why not reveal them here?" asked Roger Chillingworth, glancing quietly aside at the minister. "Why should not the guilty ones sooner avail themselves of this unutterable solace?"
"They mostly do," said the clergyman, griping hard at his breast, as if afflicted with an importunate throb of pain.
Ironically, although Hester's "A" on her breast cannot be covered up, and that very fact makes her suffering lighter in the long run, Dimmesdale's is covered, and that just increases his suffering. (This is based on the unproven assumption that he also had an "A" on his breast).