When the male higher-ups in town politics and religion meet at Governor Bellingham's house to discuss what should be done with Pearl in order to give her the best opportunity to have a productive and godly life, Chillingworth's chief interest in the conversation has to do with learning as much as possible about Pearl's father so that he can proceed with his plan to identify and torture the man with his sin. The narrator says, during this conversation, that Hester
turned to the young clergyman, Mr. Dimmesdale, at whom, up to this moment, she had seemed hardly so much as once to direct her eyes.-- "Speak thou for me!" cried she [....]. "Thou knowest, -- for thou hast sympathies which these men lack! -- thou knowest what is in my heart [...]."
Such an impassioned plea, especially when Hester had essentially ignored the minister's presence until now, cannot fail to capture Chillingworth's attention. Further, Dimmesdale speaks on Hester's behalf with such feeling that Chillingworth cannot help but mark his "'strange earnestness.'" By the end of the interview, Chillingworth likely feels that he has a clue as to Hester's co-sinner's identity, having observed their odd interactions. His only concern at this point, or any point in the novel really, is figuring out who has harmed both Chillingworth and Hester and then torturing the man with his own guilt.