This is probably one of the most obscure and puzzling names of any literary work I know of. As with anything else, research online will give us a basic explanation, but in my view this in itself doesn't totally account for why Mamet used the word as the title of his play (and film).
In the story, there is a confrontation between a college professor, John, and one of his female students, Carol. She comes to him after class one day, asking for help on an assignment. John immediately begins talking down to her, or rather, talking at her, deliberately using terms most students wouldn't understand and generally giving the impression that he has no time for her and that he's amused by her inability to grasp what he's talking about. Her frustration with him just eggs him on to continue in the same vein, lording it over her and reveling in this power dynamic of teacher-student and male-female. When she later brings harassment charges against him, he's stunned, supposedly not understanding that he's done anything wrong.
As one can see from online lookups, in its original usage "Oleanna" was a Norwegian folk-song about an ideal community in the US where immigrants from Europe were settling. The folk-singer Pete Seeger much later translated it and made it into a song naming a series of faraway, semi-mythic places where a traveler finds himself.
Mamet's use of the name probably has multiple implications. First, it suggests that the academic environment, which here is a microcosm of male-female relationships, is just as bizarre in its way as these distant and even alien places named in the song. Or, conversely, the otherworldly settings are an ideal one wishes to create or to escape to as an alternative to the dysfunctional setting of his play. The title seems to have an evocative power all by itself even if one does not know exactly what it means. The lack of obvious meaning almost enhances, paradoxically, our sense of the story's power, because the action of the play, though true to life, points out the absurdity behind such situations in the academic and the business worlds in which people (mostly men) can use their power to put down or humiliate their "underlings." These demonstrations of power are incomprehensible in some way, and it takes a stretch of the imagination to figure them out, to understand the real reason they occur. The unrecognizable term Oleanna is itself a metaphor of the incomprehension with which we're confronted.