The key to freedom is through books.
The library represents freedom outside the walls of Shawshank. It is intellectual freedom, and it gratifies Andy to turn Brooks' old musty book room into a thriving multi-media showcase. According to the screenplay, it used to be "Brooks' private domain." Andy transforms it into a public one, representative of freedom and democracy, the very opposite of prison. Prison has Andy's body, but he will not let it have his mind.
The library becomes Andy's office. It's where he does the tax returns for the guards. It becomes a meeting place not only for prisoners, but for the guards, and the guards of other prisons. It is a communal meeting spot, the antithesis of the Warden's secretive office with its locked door and hidden safe.
The library episodes serve as a turning point in the novel: while building up the library, Andy tries to play by the rules and work within the system as a "legitimate" inmate; but after Tommy is killed, Andy realizes he must think like a criminal in order to implicate the Warden and escape.
Again, Andy's key to freedom in through books--not the books in the library, but through the Warden's books: his ledger and Bible. He hides the rock hammer in the latter, and on the night of his escape switches the two. Once out of prison, he mails the ledger to the newspaper. After the story is released and indictments made, Hadley becomes an inmate and the Warden commits suicide.
The library is personified in Tommy. Tommy could have been a material witness in Andy's re-trial. He is like a book himself, as he has the knowledge to set Andy free. After the Warden kills Tommy and dismantles the library, Andy...
effectively begins plotting his revenge.
It's also a place of imagination. It's where Andy does his best work. In the library, he is compared to Rembrandt. Remember the conversation he has with Red?
RED: You can't just make a person up.
ANDY: Sure you can, if you know how the system works, and where the cracks are. It's amazing what you can accomplish by mail. Mr. Stevens has a birth certificate, social security card, driver's license. They ever track those accounts, they'll wind up chasing a figment of my imagination.
RED: Jesus. Did I say you were good? You're Rembrandt.
ANDY: It's funny. On the outside, I was an honest man. Straight as an arrow. I had to come to prison to be a crook.
The library had the potential to be a transformative place: Tommy earns his G.E.D. there, and as a place where liberal arts flourish, it stands in sharp contrast to the Bible-thumping conservatism and censorship of Shawshank. Even after the library is dismantled, Andy learns from its symbolic hope. It sparks his imagination and sets him free.