Dimmesdale is haunted by his sin. In chapter eleven, the narrator tells us that "Mr. Dimmesdale was thinking of his grave...[questioning] himself whether the grass would ever grow in it, because an accursed thing must there be buried!" The narrator continues to tell us that the public's response to him was torturous. Dimmesdale valued truth and he felt he was empty of any value, and couldn't stand that his parishioners didn't know that. He had a desire to speak out and tell them but when he tried, they worshipped him all the more thinking he was just being humble.
As the novel progresses, Dimmesdale is increasingly guilt stricken. He confesses to his congregants that he is vile, sinful, and not worthy of their respect. They respond by only holding him in higher esteem. They feel he is only showing great humility, instead of feeling he has committed a sinful act.