This answer is a play on words meant to assuage little Pearl's curiosity. The "black man" Pearl is talking about is the devil. But Hester is not talking about the devil. She is talking about Dimmesdale. As a minister, he wears black all the time. He left his mark when they committed adultery, most likely in the forest, a symbol for freedom and the one place that Hester removes the scarlet letter later in the novel. Her mark was made public when she became pregnant and gave birth to Pearl. But that discussion would be too advanced for young Pearl, and so Hester simply says the "A' is the mark of the black man. This is Hester's way of telling Pearl the truth in a way she can understand it and later, when Pearl is older, Hester can go back and explain the full meaning.
This quote comes from an interesting situation. Pearl had overheard the story of the black man. She heard the story that thousands and thousands had met the black man and signed his book. She heard that the Scarlet Letter was the black man's mark on her, and that it glowed in the dark when she went to meet nightly. Hester asked Pearl if she had ever awoken in the morning to find her mother missing. When Pearl says that she had not, Hester then makes the admission that she did indeed meet the black man and that the scarlet letter is his mark. But a mark of what?
The Letter is an external sign of the "sin" that Hester has committed. She wears it not because of her sense of personal sin, but because she has violated a rule of their society and accepts the fact that they have the right to judge her. She understands that the black man comes in two versions, personal and communal "sin." The second is the black man she has met, just once, and for that she wears the mark of social sin.
This quote by Hester about the Black man is very significant because it implies a connection between others and her sin. This is a confession by Hester to her child, her first admission of the scarlet letter signifying anything bad, or connected to another. This admission by Hester prepares Pearl to make the later connections between her mother and Dimmesdale.