Pronouns can sometimes be confusing—until you become more comfortable as to how they function in the English language.

Defined, a pronoun is...

…a pro-form that substitutes for a noun...such as...the words it (substituting for the name of a certain object) and he (substituting for the name of a person).

In other words, a pronoun is a word that takes the place of another word. This is often done to make writing more interesting, by avoiding repetition. For example, if you are speaking about Spot, you might refer to him as "the dog," but to avoid continually calling him by name or as "the dog," you might also refer to Spot as "he" or "him." (If one is unsure of the gender of the subject being described, "it" would be used.) The pronoun also follows the "number" of the item being discussed. It depends upon the subject—if singular, then "he," "she," or "it" would be used. If plural, then "they" or "them" would be used—depending upon how the pronoun functions (the "part it plays") in the sentence.

There are rules that govern the use of pronouns, as summarized above. It is helpful to know first who or what is being referred to when using the pronoun. If we speak of Jane, then further reference to Jane could be made using a pronoun: she. Jane becomes the antecedent, or the original object that the pronoun refers to. However, if Jane has not be named, using "she" can be confusing without a referent point being provided. (These kinds of pronouns are called "unprecursed pronouns.") So it is important that at some point Jane is identified so the identity of the object the pronoun refers to is clear.

With all of this said, in your example, you are referring to an "objective pronoun." Knowing when to use "he" as opposed to "him" comes down to basic rules. "Him" is an "object/target pronoun." Pure and simple...

...the target is the object on which the verb acts...

"He" is known as the "subject/initiator pronoun. This is a little easier to understand...

...the initiator is the subject performing the action.

Basically, the initiator pronoun is the subject that is doing something.

Your sentence makes it clear that someone is "volunteering." This is an action. The choice is that Mattie or the other subject of the sentence is acting. So your sentence must read...

Mattie, rather than he, volunteered to swim.

You can always double-check yourself. If you remove Mattie, "he" would volunteer. "Him" would never volunteer—or do anything else. Where there is action, choose the "initiator" pronoun.

What comes after the phrase, 'rather than' is always grammatically equivalent to what is before the phrase.  In your case, the subject of the sentence is before the 'rather than.'  So the writer would choose a pronoun that is a subject pronoun. 

The answer to this question is 'he.' The reason why 'he' is a better choice for this sentence than 'him' is because of the nature of the pronouns. Some pronouns are subject pronouns: I, we, they, he; these subject pronouns are used at the beginning of the sentence as the subject.   Some pronouns are object pronouns: me, us, them, him.