Chapter 21 marks the festivities of the English Holiday in this small Puritan colony. Several strangers are bustling about the square, including many sailors from a Bristol ship. Hester and Dimmesdale have made plans to board this very boat together in order to leave the colony, and all of their secrets, behind.
At the end of the chapter, Hester learns in a brief conversation with the ship's captain that "one more berth" must be made ready, because Chillingworth, the physician, has also secured a spot on board.
This information horrifies Hester. In that moment she catches Chillingworth's eye and he smiles at her:
A smile which--across the wide and bustling square...conveyed secret and fearful meaning.
This smile signifies that Chillingworth is aware of all of Hester's secrets. In addition to her own adultury, which is public knowledge, Chillingworth is also aware of her partner in crime (Dimmesdale), which is not public knowledge. He has been "treating" Dimmesdale as a physician, and in doing so, vengefully heaping guilt and shame upon the minister from a dark place of deceipt.
It is in the moment and in this smile, however, that Hester is finally made aware that what she has only suspected about Chillingworth's capacity for evil, is actually true.