There are several different answers to your question. Here's the one I like best.
As you read the story, you'll notice that almost nothing is certain. Hawthorne uses ambiguity is almost all cases when something strange is happening. Here are some examples:
As nearly as could be discerned, the second traveller was about fifty years old, apparently in the same rank of life as Goodman Brown, and bearing a considerable resemblance to him, though perhaps more in expression than features
This, of course, must have been an ocular deception, assisted by the uncertain light.
So saying, he threw it down at her feet, where, perhaps, it assumed life, being one of the rods which its owner had formerly lent to the Egyptian magi. Of this fact, however, Goodman Brown could not take cognizance.
When we get to the end of the story, we discover that we don't even know if ANY of it happened; in fact, it is suggested that Brown fell asleep in the woods and that the whole experience was a dream.
So if it happened it might not have happened, and it might not have happened at all :)
What is important is that Brown exits the woods with NO sense of ambiguity, with the conviction of all those he trusted since his youth are hypcrites and sinners. Of course they're sinners, but hypocrites? Only to Brown whose holds people up to an impossible measuring stick.
"The enemy of the good is the perfect."