The ministers appear in the community to hold a higher social position than Hester because they are deemed by the community to be in charge of religious and moral development. Remember that this is a Puritan community, so the people believe that they need to live their lives strictly by their communal interpretation of the Bible and its teachings. As a result, the ministers are in charge of delivering the word of God and for interpreting the Bible for the people's daily lives. Further, they are in charge of upholding God's law and seeing that any who break God's law are duly punished. While it is true that Dimmesdale is presented as a weak man, the reader must remember that dramatic irony is at play--the reader, Hester, and Chillingworth know that Dimmesdale is guilty of having an affair, but the other people in the community do not know this fact. Therefore, to the majority of the people, Dimmesdale is still seen as a paramount figure in town, while they see Hester as a condemned sinner.
On the enotes.com study guide for the Scarlet Letter you can find in depth details about the characters.
Arthur Dimmesdale is the young, charismatic minister with whom Hester commits adultery. Unlike Hester, who bears the child Pearl by their affair, Dimmesdale shows no outward evidence of his sin, and, as Hester does not expose him, he lives with the great anguish of his secret guilt until he confesses publicly and soon after dies near the end of the novel.
Dimmesdale is presented as a figure of frailty and weakness in contrast to Hester's strength (both moral and physical), pride, and determination. He consistently refuses to confess his sin (until the end), even though he repeatedly states that it were better, less spiritually painful, if his great failing were known. Thus Dimmesdale struggles through the years and the narrative, enduring and faltering beneath his growing pain (with both the help and harm of Roger Chillingworth), until, after his failed plan to escape to Europe with Hester and Pearl, he confesses and dies.
Hester Prynne is the central and most important character in The Scarlet Letter. Hester was married to Roger Chillingworth while living in England and, later, Amsterdam—a city to which many English Puritans moved for religious freedom. Hester preceded her husband to New England, as he had business matters to settle in Amsterdam, and after approximately two years in America she committed adultery with the Reverend Arthur Dimmesdale.
The novel begins as Hester nears the end of her prison term for adultery. While adultery was considered a grave threat to the Puritan community, such that death was considered a just punishment, the Puritan authorities weighed the long absence and possible death of her husband in their sentence. Thus, they settled on the punishment of permanent public humiliation and moral example: Hester was to forever wear the scarlet letter A on the bodice of her clothing.
While seemingly free to leave the community and even America at her will, Hester chooses to stay. As the narrator puts it, "Here, she said to herself, had been the scene of her guilt, and here should be the scene of her earthly punishment; and so, perchance, the torture of her daily shame would at length purge her soul." According to this reasoning, Hester assumes her residence in a small abandoned cottage on the outskirts of the community.
While the novel is, in large part, a record of the torment Hester suffers under the burden of her symbol of shame, eventually, after the implied marriage of her daughter Pearl and the death of Chillingworth and Dimmesdale, Hester becomes an accepted and even a highly valued member of the community. Instead of being a symbol of scorn, Hester, and the letter A, according to the narrator, "became a type of something to be sorrowed over, and looked upon with awe, yet with reverence too." The people of the community even come to Hester for comfort and counsel in times of trouble and sorrow because they trust her to offer unselfish advice toward the resolution of upsetting conflict. Thus, in the end, Hester becomes an important figure in preserving the peace and stability of the community.