The Merriam-Webster Dictionary defines "Irony" as:
1. The use of words that mean the opposite of what you really think, especially in order to be funny.
2. A situation that is strange or funny because things happen in a way that seems to be the opposite of what you expected.
In "The Possibility of Evil," the source of the irony is Mrs. Strangeworth as she sends out her letters to the residents of her town. Her advice and counsel is "for their own good," so that her intentions and the results strike a chord of dissonance in the reader.
It never seems to enter her mind that her unwillingness to sign her letters might be an indication that what she herself is doing is wrong. The desire to hide, which is usually considered a "red flag," creates no concern for her.
Such a "lack of self awareness," as per your question, does indeed contribute to the irony.