Prospero tells Miranda that when he was expelled from his kingdom, Gonzalo made sure that he was provided with clothes and food, and also gave him his books.

Knowing I loved my books, he furnish'd me
From mine own library with volumes that
I prize above my dukedom. (Act 1, Scene 2)

The books are what give Prospero his magic. He is a wizard of sorts.  In addition to communing with magical creatures like Ariel, Prospero conjures up the gods Iris, Juno, and Ceres to entertain his daughter and Ferdinand at their wedding.

Caliban says that Prospero is powerless without his books.

Remember
First to possess his books; for without them
He's but a sot, as I am, nor hath not
One spirit to command: they all do hate him
As rootedly as I. Burn but his books. (Act 3, Scene 2)

Prospero is intent on revenge at first.  He causes the shipwreck and then has Ariel trap them in an enchantment. She calls them “men of sin,” referring to Antonio, Alonso, and Sebastian.  They conspired to have Prospero ousted from his kingdom.  Propsero has a hard time forgiving his brother Antonio especially.

However, Prospero decides not to take revenge against them.  Ferdinand and Miranda are happy.  Prospero has been on the island long enough, and now has a chance to get off.  He decides to throw away his books and take back his kingdom.

But this rough magic
I here abjure, and, when I have required
Some heavenly music, which even now I do,
To work mine end upon their senses that
This airy charm is for, I'll break my staff,
Bury it certain fathoms in the earth,
And deeper than did ever plummet sound
I'll drown my book. (Act 5, Scene 1)

While magic means a lot to Prospero, family means more.  He loves his daughter.  He is thrilled with the chance to return to his kingdom.  Prospero decides that he has spent enough years in bitterness, and forgiveness is the way forward.