His secret poison is that he has violated who he is; he has not just commited a forbidden act as is the case with Hester. He spends the rest of his life after the act knowing that he has violated his nature, but unable to acknowledge it because of his need to continue as a minister. As with "The Minister's Black Veil," Dimmesdale's acquaintance with evil ironically makes him a better minister.
That being said, I think the quote you cite is about Chillingworth and not Dimmesdale, and refers to the hatred that consumes him and that all but oozes out of him when in Dimmesdale's presence. Let me know if I read this incorrectly.
Dimmesdale's secret poison is that he has not confessed his sin. It is bad enough that he committed that sin, but even worse, that he has allowed Hester to bear the burden of it all alone. He is as guilty as her, and has not confessed, and it eats away at him. He feels himself a hypocrite, a liar, a coward, and unworthy of his role as Reverend. He constantly pits his fear of exposure against the disgust this fear gives him. It eats away at him so intensely that it eventually leads to his demise.