In Chapter 4 of The Scarlett Letter, after Hester's public punishment on the scaffold of public shame, she is in such a state of wild distress once she is returned to her prison cell that the jailer is compelled to call in a physician. The physician is the same man from the crowd whom Hester had seen earlier who had riveted her eyes in a fixed gaze of recognition:
appeared that individual, of singular aspect whose presence in the crowd had been of such deep interest to the wearer of the scarlet letter.
Although, by a seemingly careless arrangement of his heterogeneous garb, he had endeavoured to conceal or abate the peculiarity, it was sufficiently evident to Hester Prynne that one of this man's shoulders rose higher than the other. Again, at the first instant of perceiving that thin visage, and the slight deformity of the figure, she pressed her infant to her bosom with so convulsive a force that the poor babe uttered another cry of pain.
Though no direct statements are made, the stranger, calling himself Chillingworth, fits the brief description given in Chapter 2 of the intellectual scholar who was her husband whose aspect floated before Hester's eyes along with that of the long absent visages of her father and mother. With this in mind, Hester's remark to him upon his offer of a medicinal for her baby identifies Chillingworth as someone known before, in fact, it identifies him as her long missing husband.
Hester repelled the offered medicine, at the same time gazing with strongly marked apprehension into his face. "Wouldst thou avenge thyself on the innocent babe?" whispered she.
Along with the fact of his singular physical description marking him as one whom she knew before, Hester's use of the word avenge confirms that he was in fact her husband for it is only her husband who would have cause to avenge himself of her crime of adultery. There is further confirmation of his previously known identity in the narrator's remark that Chillingworth gazed at Hester with:
a gaze that made her heart shrink and shudder, because so familiar, and yet so strange and cold.