I can only suggest that it is irrelevant; the effect on Brown of what happened is all that matters. Much like Brown, we all must take a journey "into the woods"; we all must learn that everyone is made up of some good or some evil. How we deal with that knowledge is what is important. Brown can not (or choses not to) deal with what he learns. He wants people on his terms; he has developed a certain expectation of his catechism teacher, his pastor, his wife, and that is the way that he will take them ... no other.
This is a recurring theme in Hawthorne. If you have read "The Scarlet Letter," you know that all of the characters have some good and some evil in them, that they (we) are all complex characters. The society of TSL, much like Brown, fails to accept this and punishes only one aspect of a person. Even the threat of this punishment wrecks Arthur's life.
This theme of accepting only the perfect, and its destructive effects on individuals appears in many other Hawthorne works: "The Birthmark," and "The Minister's Black Veil" are other examples you might want to pursue.
"The enemy of the good is the perfect."
There is not a definitive answer to your question! The ending of the story was purposely left ambiguous. It is up to the reader to decide if it was all just a dream or if it really happened.
What is clear is that Brown was changed by the dream/event and became an embittered man. He died an unhappy and untrusting person. This is a sharp contrast to the happy and confidant man that took went on a walk with a stranger.