Hester and Pearl leave Chillingworth to his own devices. Seeing him bent over, collecting what may well be poisonous herbs, Hester finally admits to herself that she hates the man. She can't remember why she married him. Meanwhile, Pearl has been entertaining herself in the shallows, building boats out of birch-bark and playing with the starfish ("five-fingers"). Her often violent nature reveals itself yet again when she begins throwing rocks at a flock of birds. This amuses her until she realizes she's injured one of the birds, potentially breaking its wing.

Tired of her games, Pearl decides to fashion herself a headdress out of seaweed. She makes an entire mermaid costume, finishing it off with a green letter "A" mimicking her mother's scarlet one. Hester chastises Pearl, asking the girl if she understands why her mother has been forced to wear the scarlet letter. Pearl says it's the same reason Dimmesdale puts his hand over his heart, but when pressed she can't explain what she means. This leads Hester to ruminate, once again, on Pearl's strange behavior. Even her displays of affection is hard to predict, and Hester never knows when Pearl will be kind to her and when she'll ignore her entirely. She loves her daughter anyway.

Pearl asks her mother point blank what the scarlet letter means. She asks again and again, but Hester refuses to explain.