In the Puritan society that Hester belongs to, the forest is already rumored to be full of witchcraft and spirits. The forest is a symbol for secrets/darkness/evil to the people of the settlement.
However, the forest is also the only place that Hester can be completely honest and herself with Dimmesdale. In the town, she can treat him only as her reverend and not as a familiar friend. Within the protection of the forest, Hester is free to talk to Dimmesdale like a friend and even the lover he had been. She can more openly discuss Pearl, her feelings, and anything that has been on her mind.
The town is out to find the man who impregnated Hester and have him share in her punishment, so it is more difficult for them to have any sort of personal exchange inside the settlement. Within the confines of their society, they are forced to act like near strangers. He must treat Hester like he treats anyone else in the town. In the forest, however, the reality of their sin can be let out in the open and Dimmesdale is also able to disclose his real feelings and suffering to Hester.
Beyond that, Hester resolves to tell Dimmesdale who Chillingworth truly is and she can only disclose such information out of the earshot of anyone in the settlement.
It would seem that Hester is trying to protect Dimmesdale from being exposed as the father of their child. She knows she is under especially close scrutiny in that small New England settlement. It is best if she is not seen talking to Dimmesdale at all. In any case, they certainly cannot talk freely inside the settlement. They never know who might be watching from what vantage point. Part of the pathos of this novel is involved in the fact that she protects Dimmesdale from exposure and he allows her to do so. He realizes that he is behaving in a shameful manner. He probably suffers more than Hester because she has been exposed and humiliated, whereas he not only has remain figuratively in hiding but has to see her being publicly shamed and abused. It must be a relief to him to be able to meet her in the forest where no one can see them. At least he can drop the painful mask he wears most of the time.