The answer to this question can be found in Chapter Five of this novel, where Hester's life after she leaves prison with her daughter is introduced to us. Hester's new life is very symbolic in terms of where she choses to situate herself, as the following description makes clear:
On the outskirts of town, within the verge of the peninsula, but not in close vicinity to any other habitation, there was a small thatched cottage. It had been built by an earlier settler, and abandoned because the soil about it was too sterile for cultivation, while its comparative remoteness put it out of the sphere of that social activity which already marked the habits of the emigrants.
It is this "small thatched cottage" that Hester chooses for her dwelling. The fact that it lies on the fringes of the town and civilised life is particularly fitting, as Hester, through her act of adultery and the scarlet "A" she is forced to wear on her breast, is forced to operate on the fringes of respectable society. Her living place therefore seems apt based on her position amongst the Puritans who have shunned her. That the location has "sterile" soil is again important as it reflects Hester's position in society. She is only free to live somewhere without any perceived value as befits her status as sinner and outcast.
When Hester Prynne is released from prison with her infant daughter, she does not leave New England, as the father of her child (who is not yet known to the town) lives nearby. Instead, she chooses to live in a remote thatched cottage at the edge of the town, where the quality of the soil makes it difficult to grow crops. The cottage is near the shore, and it has a view across the sea of forest-grown hills. A clump of scrubby trees grows outside the cottage, as if to mark it as a suspicious spot.
At the cottage, Hester grows a garden and supports herself with her fine needlework skills. She is a talented seamstress, and her scarlet letter "A" is almost an advertisement for how fine her work is. Hester and her daughter, Pearl, live in social exclusion, and Pearl, who begins to resemble a kind of woodland sprite, has the trees and flowers as playmates, as no one in the town visits their lonely cottage.