As your question implies, Goodman Brown's journey to the dark side is both physical and mental--in other words, Brown's desire to experience evil leads him into a physical space that no Puritan villager would normally go alone:

He had taken a dreary road, darkened by all the gloomiest trees of the forest. . . . and there is this peculiarity in such a solitude, that the traveler knows not who may be concealed by the innumerable trunks and the thick boughs overhead . . . he may be yet passing through an unseen multitude.

Given the real dangers of the forest--Indians being the most immediate--a Puritan villager in his right mind would never inter the forest without weapons and other villagers for protection.   Goodman Brown's dark urges--partly a reaction to his strict, conventional Puritan beliefs--are made physical by the forest itself, a gloomy collection of trees so thick that they seem to close immediately behind him.  In a very real way, Goodman Brown's walk into the forest is a dark journey into his own mind.  And what makes this journey so unusual is that Goodman Brown willingly walks into a real environment in which evil often comes in the form of Indian attacks.

Because one of Goodman Brown's Puritan beliefs is that mankind is inherently sinful, his mind is primed to see this belief made real.  When he meets Satan and, later, sees Goody Cloyse, Deacon Gookin and the minister, seeing these people, whom he trusted with the health of his soul, in the forest--a symbol of darkness and evil--Brown becomes almost convinced that his journey is justified.  A few minutes later, when Brown thinks he hears Faith's voice overhead, and then sees her pink ribbon fall to the ground, his Puritan world is finally and completely turned upside down.  His faith in his wife, his religious leaders, his society is gone, and his faith in God, which has been undermined by everything he encounters (or dreams he encounters)  in the forest, is the last to go.

That Goodman Brown's conversion to the dark side is complete is demonstrated by his rush through the once-dreaded forest, which now holds no danger for him:

. . . leaving him in the heart of the dark wilderness, still rushing onward with the instinct that guides mortal men to evil.  The whole forest was peopled with frightful sounds. . . .

The forest, which would normally hold nothing but terror for a Puritan young man, now is merely an extension of Goodman Brown's state of mind.  Because he has become convinced that everyone is evil, the evil of the forest is no longer particularly threatening.  At least until the climactic scene in which Brown tries to save his wife, Faith, the forest is Brown's new home.