This is an interesting question, because if we look at the poem carefully I am not too sure that the speaker is merely following a "whim" as you describe it in your question when he states his desire to go back to the sea. If we have a look at the way he describes his desire, it is almost as if it were a compulsion rather than something as paltry as a "whim." Consider the way that each stanza begins with the opening refrain: "I must go down to the seas again..." This speaks of something that is somewhat stronger than a mere whim. Clearly, the speaker is feeling incredibly restless and intensely desires to return to the sea, however I would argue that the "Sea Fever" that he suffers from is actually something like a compulsion rather than a mere idea or capricious thought.
As to whether the speaker is "right" to give in to this compulsion, I think the poem clearly depicts above all the deep and enduring love that the speaker has for the sea. The sea, to the speaker, seems to be life itself, and he longs to feel alive in the way that he is only able to when he is out on the sea:
And all I ask is a windy day with the white clouds flying,
And the flung spray and the blown spume, and the sea gulls crying.
Therefore, I don't see that there is anything wrong with desiring to see and experience again what your heart and your whole being cries out for, in the way that the speaker obviously longs for his return to the sea. His desire to return will obviously impact nobody negatively, and so I think he should be free to follow his heart's desire.