Starbuck understands two things about Ahab's quest that lead him to protest. He sees that Ahab is bent on overthrowing the natural order (and defying God in the process) and he recognizes the extreme danger that this pursuit will bring to the crew.
Starbuck is the most devout member of the crew and also the senior officer. It falls to him to bring the challenge to Ahab, which he does on these grounds.
Starbuck's protests are overwhelmed by the maniacal Ahab in the end.
In the soliloquy, “Dusk,” Chapter 38, Starbuck explains to himself that Ahab has “drilled deep down, and blasted all my reason out of me! I think I see his impious end; but feel that I must help him to it.”
Though Starbuck feels that it is illogical to take revenge on an unthinking animal that acted on "blindest instinct", he is not able to convince Captain Ahab of this perspective. He is also not able to match or truly challenge Ahab's dedication and resolve.