The part of the novel you need to re-visit to answer this question in Chapter 11, entitled "The Interior of a Heart." This is rather a sinister chapter as it talks about the way that Chillingworth plans to psychologically encroach on Arthur Dimmesdale's privacy and space, probing into the very centre of his soul to discover what deep and dark secret oppresses him so badly.
However, as you indicate, the severe impact of Dimmesdale's guilt does have an impact on his work as a minister. Consider what the text tells us:
While thus suffering under bodily disease, and gnawed and tortured by some black trouble of the soul, and given over to the machinations of his deadliest enemy, the Reverend Mr. Dimmesdale had achieved a brilliant popularity in his sacred office. He won it, indeed, in great part, by his sorrows. His intellectual gifts, his moral perceptions, his power of experiencing and communicating emotion, were kept in a state of preternatural activity by the prick and anguish of his daily life.
It is his guilt that allows Dimmesdale to have "sympathies so intimate with the sinful brotherhood of mankind." "His heart vibrated in unison with theirs," we are told, making him accessible. Part of his success, then, lies in the fact that he himself identifies himself as a sinner and recognises the truth of that fact in his own life. This gives him no arrogance or pride when he ministers to those who are like him. Such a manner and approach to people in spiritual need endears him to them, making him immensely popular.