I believe you are referring to the incident in Chapter 20, "The Leech and his Patient." Here, along with her mother, Pearl meets Chillingsworth and Dimmesdale together:

...the child laughed aloud; and shouted, --"Come away, mother! Come away, or yonder old Black Man will catch you! He got ahold of the minister already...".

Although Pearl is ostensibly addressing her mother, she is also directing her barbs towards the two men, first letting Chillingsworth know that she, for one, is on to him, and secondly, that Dimmesdale has succumbed to the "black man," ie, the dark side. So, yes, Pearl is correct.