Ironically, Dimmesdale's guilt makes him more of a popular minister than he already was. Of course, his guilt is known only to himself, but it does manifest itself in a number of ways which only serve to make him more admirable.
For example, the more he declares to be an unworthy spiritual leader, the more the protests grow to the contrary. The public simply sees this as a self-effacing characteristic by a most humble minister. The more he tries to proclaim his guilt in passive ways, the more the public sees him as a truly righteous leader.
In addition, his appearance and mood becomes more solemn and introverted. He mopes around, becomes pale, talks socially infrequently and appears to becoming physically ill to the point that he obtains a private physician. However, all of these traits just reinforce the public's idea that he is willing to sacrifice everything for God.
Only when the physician's mental pressure and his love for Hester drive him to the decision to leave with her, does his true guilt appear to all. However, he dies, making himself forever a martyr in the intolerant community.