According to the story, the townsmen accepted and respected Chillingworth because he was educated, and because he showed an accentuated interest in the man whom the town loved and admired the way they would venerate God himself. Yet once Chillingworth' true intentions began to take place (which was to find Dimmesdale out as his wife's lover), his action spoke louder than words.
At first, his expression had been calm, meditative, scholar-like. Now, there was something ugly and evil in his face, which they had not previously noticed . . . it grew to be a wisely diffused opinion, that the Reverend . . . was haunted either by Satan himself, or Satan's emissary, in the guise of old Roger Chillingworth. (1224)
Therefore, the townspeople felt that Chillingworth was haunting Dimmesdale, but what they do not know is that the reason why Dimmesdale is allowing himself to be haunted is because he is guilty, and because Chillingworths other mission was basically to drive Dimmesdale to insanity so he would confess who he really was.