Just as in real-life situations, people respond to events and circumstances according to their prior experiences. When, in The Scarlet Letter, the letter "A" seems to appear in the sky, the sexton declares it means "Angel," symbolizing Governor Winthrop's recent death, yet characters like Reverend Dimmesdale shudder because for them, the letter "A" symbolizes the greatest sin they have committed -- adultery. Even today, people like you and I might see a "sign" (say, for example, a penny turned heads-up) and interpret it differently -- you might pick it up for good luck, but I might just look at it with guilt because I recently passed a homeless man asking for change. Because of their personal experiences, the characters in The Scarlet Letter react in much the same way, according to the events and character traits that have shaped them.
Nathaniel Hawthorne, who also serves as narrator in The Scarlet Letter (as we can tell by his preface, "The Custom- House"), deliberately remains ambiguous about what the "A" does mean. Even at the end of the novel, when Dimmesdale reveals the "A" carved on his chest, and some townspeople say they see the emblem, but others do not (and some even concoct stories about how it possibly came to be), Hawthorne claims, "The reader may choose among these theories." When, in "The Custom-House ," Hawthorne explains how he came to write about this scarlet letter, he describes the actual scarlet letter he found and touched as producing a sensation "as of burning heat." Certainly, many readers find Hawthorne sensitive to the plights of Hester Prynne and Arthur Dimmesdale, possibly through his limited omniscience into those characters' lives. His sympathies for our protagonists, however, might also stem from a deep-seated guilt for one of his ancestors' -- John Hathorne's -- active persecutory role during the witch trials.
Overall, I would say that Hawthorne explains these two different interpretations through the characters' past experiences and reactions, though Hawthorne, as narrator, never actually explicitly states anything at all about their two differing responses to the "A". It is easy to support the fact that these two men responded so differently to the "A" because their inner guilt (or lack thereof) and their knowledge of deep sins (or lack thereof). Throughout The Scarlet Letter, Hawthorne leaves much of the decision-making to his readers; what is or is not real are constructs in our minds.