What does "abject" mean as it is used in the sentence?
As it's used in the story, "abject" means "desperate" or "despicable and low."
Let's take a look at it in context:
"After all, I had already fallen in love with the savage brat, the parents were contemptible to me. In the ensuing struggle they [the parents] grew more and more abject, crushed, exhausted while she surely rose to magnificent heights of insane fury of effort bred of her terror of me."
As you can see here, the sentence is saying that the parents are losing the fight. They're getting beaten, and they're really tired. They're basically sinking down into a desperate, despicable struggle as they try to get their daughter to comply with the doctor's throat exam. At the same time, Mathilda herself is doing the opposite of what her parents are doing: they're sinking down low and losing, but she's rising up high and winning, temporarily keeping the doctor out of her throat.
So, the Olsons are abject: they're low, despicable, and absolutely desperate in this situation.
"Abject" can also mean "utter or total" in other contexts, but as this one makes it clear, the parents are abject in the sense of "low and desperate."
You can remember "abject" and its meaning by understanding its roots and by comparing it to words you already know that have "ject" in them. "Ab-" means "down" or "away," and "ject" means "throw." So, something abject is "thrown down" or "thrown away" in the sense that it's very low down. Compare that to what you already know about "inject" ("thrown in,") "reject" ("throw out" or "throw back") and "eject" ("throw out.")