I don't see Ishmael as either the protagonist or the antagonist in Moby Dick. Ishmael is the narrator, and as such he reminds me of Nick Carraway in The Great Gatsby. It is Captain Ahab whose motivation drives the story. His motive is to kill the whale, so I consider Moby Dick the antagonist. Ishmael is just along for the ride, so to speak. He has to go wherever Ahab decides to go. In The Great Gatsby, set in quite a different time and milieu, Nick Carraway seems to get carried around a lot by both the protagonist, Gatsby, and the antagonist, Tom Buchanan. What is the advantage in telling a story through this "minor character as narrator" technique? It imposes certain limitations on the author, but at the same time it gives the author more freedom to express his feelings, impressions, and opinions through his narrator than would otherwise be the case if he wrote his tale as a straight, conventional omniscient- third-person narrator. Also, the reader might find it easier to identify with a more modest character like Ishmael or Carraway, than with a more epic type like Ahab or Gatsby. Ishmael and Carraway both survive the conflicts, while Ahab and Gatsby do not. Therefore, Ishmael and Carraway can tell the complete stories, while Ahab and Gatsby both get killed. The stories are presented as learning experiences for both narrators, not necessarily learning experiences for the protagonists. What Ishmael and Carraway experience is intended to be a learning experience for the reader as well.
This is a rather complex question, as it turns out. I agree that the whale can be seen as the antagonist. The whale is the embodiment of the forces of nature (or God) which Ahab sets out to overthrow. The whale is certainly Ahab's nemesis. If we consider Ahab to be the protagonist of the novel, the whale is clearly the antagonist (along with the forces aligned with it (the sea, God, nature, etc.).
However, if Ishmael is considered the protagonist because he is literally the primary figure in the text, appearing first and last, narrating the tale, etc., we have a harder time situating the whale as the bad guy.
If Ishmael is the protagonist, Ahab becomes the more likely candidate for antagonist. It is Ahab, after all, who leads the ship and crew to destruction and death as a result of his misguided and pride-driven quest to get revenge on a whale.
You could argue that the antagonist is the whale, or if you look at the book as a psychological struggle with one's person inner demons, the antagonist is actually Ishmael.