How would it make them look if they convicted "innocent" people and the girls were lying? They would look like fools! You know the saying from the Bible, "Pride goes before a fall"? These men took pride in who they were--they were godly, righteous, logical thinkers. If they could be taken in by a bunch of silly girls, who would ever be able to trust or respect the men ever again? Plus, the blood of innocent people would be on their heads. It's so much easier to live with one's conscience if you don't even entertain the thought that your witnesses were lying. The entire reputation of the men and the court rested on their believing the girls.
There are several instances where they refer to the law and how it "binds" them.
Act Two (where Herrick comes to arrest Elizabeth): "I have nine men outside. You cannot keep her. The law binds me, John, I cannot budge"
After that, Reverend Hale say "The court is just"
About a page after that, Giles Corey even says to Hale, "It is fraud, you know it is fraud! What keeps you, man?"
Soon after, Hale talks for a couple of paragraphs--those would really work as to why they have trouble doubting "little girls".
Look at Act Three also--where Francis Nurse and Giles Corey go to bring evidence. You can feel the haughtiness and pride in the words of Hathorne and Danforth.