In order to answer this question, you need to focus on how the night he spent in the woods, whether it was just a dream or based on reality, changed Goodman Brown, and marked the end of his young, carefree days. Consider what the final paragraph tells us about how he was transformed:
A stern, a sad, a darkly meditative, a distrustful, if not a desperate man did he become from the night of that fearful dream.
Having seen the evil side of every person that he has known and respected, from his pastor to his dear wife, Faith, Goodman Brown is now unable to accept and process this truth. The universal sinfulness of humanity that his dream demonstrated has crushed Goodman Brown, and turned him into a man who is unable to extract any enjoyment out of life whatsover. It is this central lesson that the night in the woods taught him that made him hopeless and depressed, and resulted in his "dying hour" being characterised by only "gloom."