I'm not sure if you're asking what is the scariest part of the story for Goodman Brown or what's the scariest part for the reader. If you're asking the latter question, that is certainly a personal question only you could answer. It would depend on what scares you. Remember, Goodman Brown isn't sure that his night in the forest really happened or if he dreamed it. He allows that night, whether real or a nightmare, to change his life forever.
If you're asking what the scariest part was for Goodman Brown, I would say it was when he saw his wife going to the devil's meeting in the forest. At the beginning, Goodman believes his wife is an "angel" and is one of the few who will be chosen to go to heaven. When he sees Faith, his wife, in the forest, it changes his life forever. Goodman Brown loses his Puritan faith and never gets it back. He lives the remainder of his life alienated from his wife and the rest of the people. In this respect, he becomes the symbolism of Hawthorne's belief in the isolation of the human spirit. Goodman Brown resists evil and retains his faith, but he loses all hope, living isolated from everyone.