Of course the "A" starts as "adulteress." During the course of the novel, in most traditional interpretations, it takes on other meanings that almost supercede the original intent of the letter as it comes to stand for "Able" or "Angel." An interesting interpretation which I think I have suggested elsewhere is that it stands for "Arthur." We know that Hawthorne could have named Dimmesdale Bill, or John, or Charles, or anything he wanted --- so why Arthur if it isn't related to the "A"? It adds great irony ... when they are asking her to confess, she has her partner's initial on her bosom throughout the entire interrogation. For the rest of her life she "confesses" every day, but no one can decode the symbol (something that Hawthorne is always playing around with) ... the people see it as just a sign of her guilt, we know it as a symbol of much more.
I'm not sure everyone would agree with this interpretation, but it seems pretty convincing to me.