In "The Great Gatsby", what about the library at Gatsby's house surprises Nick?
As others have expressed, it is the owl-eyed man who is surprised by Gatsby's library. The library's verisimilitude surprises him. The owl-eyed man had not expected to find real books on the shelves. He thought they would be cardboard cut-outs with spines made to look like books, but when he pulls one down, it is real.
The owl-eyed man perceives that Gatsby is a fraud, but doesn't judge him for it. In fact, he's impressed with his expertise at illusion. Owl Eyes call Gatsby a "Belasco." This is a reference that could easily be lost on a modern audience. In the 1920s, David Belasco was famous as a theater producer who worked hard to create very realistic sets for plays. The owl-eyed man is effusing about how well Gatsby has staged his life.
The owl-eyed man also notes that the pages of the books are uncut. The books are real, but Gatsby hasn't read them. The owl-eyed man seems to think Gatsby did this on purpose, but this shows that Gatsby doesn't quite have his mask on perfectly. it's also worth noting that the book the owl-eyed man pulls from the shelf that Gatsby hasn't read is the first volume of the "Stoddard Lectures," presumably the same Stoddard whose racist theories Tom Buchanan is reading when Nick goes to dinner at his house. Maybe Gatsby would have been better prepared for Tom if he'd read the book.
I assume you are referring to the scene in chapter 3 with the man in the owl-eyed glasses. It isn't Nick who is surprised; it is the man in the owl-eyed glasses. Nick and Jordan are at one of Gatsby's parties, the first one Nick attended. They go into the library where they encounter the man in the glasses. The man tells them, with surprise in his voice, that the books are real - they have pages and everything. The man goes on to say that he's checked the books and they are indeed authentic books, not cardboard. He also says that he's quite drunk. This happens after Nick has heard many contradictory stories about Gatsby and before he's actually met him. Gatsby, at this point in the story, has a mysterious, almost unreal quality about him. Even though the reader hasn't really met him yet either, enough information has been hinted at to let us know that there is something not quite true about the man. It's fitting, therefore, for the man in the glasses to suspect that much more about Jay Gatsby, including his books, is not real.
As was mentioned in the previous post, the owl-eyed man, not Nick, was surprised by Gatsby's books. In Chapter 3, Nick attends one of Gatsby's elaborate parties. During the party, Nick meets up with Jordan and the two walk through Gatsby's home to look for him. They end up entering Gatsby's massive library where they meet a stout, middle-aged man wearing owl-eyed glasses. The owl-eyed man is drunk and comments that he is surprised at the fact that the books in Gatsby's library are real. The man tells Jordan that he expected the books to be made out of durable cardboard. When the owl-eyed man takes a book off of the shelf to show Nick and Jordan that they are real, he notices that the book is "uncut." The significance of the books being "uncut" means that Gatsby has not read them which suggests that he is indeed a fraud. The books, like Gatsby, are a facade. Gatsby is trying to give the impression that he is a well-read, educated individual when he is actually a successful bootlegger.