The line that you cite is spoken by Roger Chillingworth. He says it to Dimmesdale.
The context here is that the two of them are talking with the Governor and some others. They have been talking to Hester, trying to decide whether she is a "fit mother." If they decide she is not, they will take Pearl away from her.
Dimmesdale has just said that Hester should keep the child for the sake of Hester's own soul and because that is how God has set things up.
Roger's line here is just commenting on what Dimmesdale said. He's saying that Dimmesdale seems unusually or strangely serious (earnest). This is significant because we know that Dimmesdale is actually Pearl's father and that is probably why he is so earnest on this subject.