Being the cuckhold is not a position that any man wanted to be in at the time of Hawthorne's story. In addition to this and the other points already made, Roger Chillingworth does not wish to make explanation why he has abandoned his wife to live with Indians where he has studied herbs and medicinal cures. His past history would be subject to questioning as, according to the Puritans, such actions are those of witches.
When Roger Chillingworth came into Boston after spending years in the wilderness, he found his wife standing on the scaffold holding her infant. Roger knew immediately that Hester had committed adultery in his absence. His obsession to find Hester's partner in sin began at that moment.
Chillingworth wanted no further relationship with Hester. Theirs had not been a marriage based on love to begin with. Hester had not loved him when they maried and had been honest with him about that. Chillingworth does not claim Hester as his wife because he does not want his real identity known. When he visits her in the jail after she is taken from the scaffold, he exacts her promise that she will not reveal it.
By keeping his relationship with Hester a secret, Chillingworth is then free to live in the village and begin his search for Pearl's father. He wants only one thing: revenge for the wrong that has been done him by the man he intends to find and punish.
In Chapter 3, Chillingworth gestures to Hester that she should not reveal his identity and remarks to a stranger in the crowd that Hester’s husband must have been foolish to think he could keep a young wife happy. Chillingworth’s willingness to take some of the blame for Hester’s “fall” seems almost noble. He admits that he was not the right husband for Hester and that he was remiss in not joining up with her sooner (even though he seems to have been held captive). Yet, he ultimately chooses to use his knowledge for vengeance. Chillingworth's lust for revenge is what drives fim to stay in Boston despite his wife’s betrayal and disgrace. He is a scholar and uses his knowledge to disguise himself as a doctor, intent on discovering and tormenting Hester’s anonymous lover.