In the context of the play, the religious leaders are attributed with power only in conjuction with moral authority. Hale is called a "broken minister" because he loses his moral authority in the eyes of John Proctor when he acts against his conscience.
Parris and Hale are each characterized as being devoid moral power by John Proctor, the figure who is essentially the moral authority of the play.
For Proctor, Hale loses his moral authority when he is shown evidence that clearly leads him to doubt in the truth of the claims of witchcraft made against both Elizabeth Proctor and Rebecca Nurse. At this point, despite his doubts and knowing what will happen to the women once they are accused and arrested Hale chooses to continue aligning himself with the court and the trials.
A man who will not listen to his own sense of reason and justice has no moral authority, for Proctor, and so can have no real, religious power.
In Act Two, Reverend Hale visits Proctor's home in search for evidence of witchcraft. During his visit, Proctor tells Hale that Abigail Williams told him that the children's illness has nothing to do with witchcraft. Hale's initial suspicions seem quelled until Cheever arrives with a warrant for Elizabeth who has been accused by Abigail. After Cheever finds a poppet that supposedly suggests Elizabeth attempted to murder Abigail, Mary Warren admits that she made the poppet and stuck the needle in its stomach. Hale then questions Mary by asking if someone is conjuring her to admit that she placed the needle in the poppet. However, Mary assures Hale that she is acting on her own accord. Proctor finds Mary's confession to be obvious proof that Elizabeth is not a witch and tells Hale to leave his home. Elizabeth then learns that she has been accused by Abigail and says that Abigail should be "ripped out of the world!". Cheever is quick to take notice and Proctor suddenly rips the warrant out of his hand. Proctor is furious and curses the Deputy Governor. He then calls Reverend Hale a "broken minister" and tells him to get out of his home.
Proctor refers to Hale as a "broken minister" because he believes that Hale is an unreasonable coward who does not represent God's authority. Instead of following his conscience, Reverend Hale succumbs to the hysteria and chooses to support the corrupt court. Proctor realizes that Hale is not being just and calls him Pontius Pilate as Elizabeth is taken away.