This quote comes from an interesting situation.  Pearl had overheard the story of the black man.  She heard the story that thousands and thousands had met the black man and signed his book.  She heard that the Scarlet Letter was the black man's mark on her, and that it glowed in the dark when she went to meet nightly.   Hester asked Pearl if she had ever awoken in the morning to find her mother missing.  When Pearl says that she had not, Hester then makes the admission that she did indeed meet the black man and that the scarlet letter is his mark.  But a mark of what?  

The Letter is an external sign of the "sin" that Hester has committed.  She wears it not because of her sense of personal sin, but because she has violated a rule of their society and accepts the fact that they have the right to judge her.   She understands that the black man comes in two versions, personal and communal "sin."  The second is the black man she has met, just once, and for that she wears the mark of social sin.

This quote by Hester about the Black man is very significant because it implies a connection between others and her sin. This is a confession by Hester to her child, her first admission of the scarlet letter signifying anything bad, or connected to another.  This admission by Hester prepares Pearl to make the later connections between her mother and Dimmesdale.