The fact that he is conscious that he has lived against every thing he has ever preached against by having an affair with Abigail, by knowing that he cannot just whip up a case against her to set himself free from her, and by knowing what Abigail is actually capable of doing due to the hatred she sustains against Elizabeth, Proctor's wife.

He lives in constant fear of being found out, not only because what he did was wrong, but also because it will bring consequences with it that won't be funny. For all he would speak on behalf of his impeccable reputation, he now would have to face the music in the event that all comes to the open, and the humilliation will be incredible. In the end, he was able to redeem himself and act on behalf of what was correct, and not what would make him acceptable, but-nevertheless- the imprisionment he built around himself for the sake of making an image prevented him from living a normal life.