For Caliban, Prospero's library full of magic books is the source of his power. But for Antonio, Prospero's scheming brother, it's the exact opposite: it's a source of weakness. Prospero spends so much time in his library that he neglects his duties as Duke of Milan. This allows Antonio to get involved in a dastardly scheme to usurp Prospero and have him banished to a remote island.

It's somewhat ironic that Prospero's spending all that time with his magic books didn't ultimately save him from being sent into exile. Surely he could've rustled up a magic spell or two to send his greedy, scheming brother and his co-conspirators packing? In any case, Prospero will put the fruits of his learning to good effect later on when he'll use his magic powers to whip up a gigantic storm to bring Antonio and his partners in crime to his remote island home.