Dimmsdale wears the outer garments of a priest, and these garments are supposed to be a mark of his purity and authority in the Puritan community. Yet beneath that guise, Dimmsdale is harboring the secret of his affair with Hester, and it is slowly killing him. As Dimmsdale's physician, Chillingworth manages to get close enough to Dimmsdale while he is asleep and look for the physical manifestation of Dimmsdale's inner guilt. "The physician advanced directly in front of his patient, laid his hand upon his bosom, and thrust aside the vestment, that, hitherto, had always covered it even from the professional eye (95)." We are never told what exactly what he sees, but it is presumably what he was looking for, as after he sees it he is overcome with a sense of triumph.
Later, when the relationship between Dimmsdale and Chillingworth is described in chapter eleven, it says that when Dimmsdale looked at Chillingworth, certain aspects of the man's appearance frightened him. Even "...the very fashion of his garments, were odious in the clergyman's sight; a token implicitly to be relied on, of a deeper antipathy in the breast of the latter than he was willing to acknowledge to himself (96)." Though Dimmsdale doesn't know that Chillingworth is Hester's vengeful husband, something in his clothing somehow hints at the malignant nature within.
When Dimmsdale delivers his sermons to the community, he even acknowledges that his outer garments are part of his deceit to other members of the church. "I, whom you behold in these black garments of the priesthood...am utterly a pollution and a lie (98-99)!"
Dimmsdale's clothes serve to cover his shame, while Hester's clothing, with the large A sewn to the breast, announces hers. With nothing to hide, Hester is not eaten away by guilt, while Dimmsdale's clothes constantly cover him in a false guise of purity and he is plagued by feelings of guilt and remorse. Chillingworth's appearance initially gives no hint to his ill intentions, but as his malice and thirst for revenge grow, his appearance degrades to match. The community in which these characters live places a lot of importance on reputation and public appearances. Hester admitted to her sin, and the people made her wear something that would tell everyone of her guilt, making her an outcast in the community. Dimmsdale is able to maintain his important position in the community not because of any good deeds- he is guilty of the same sin as Hester- but because he did not admit to his guilt and therefore has no mark of public shame on him. Clothes serve to contrast outer appearances with the hidden truth beneath.