This quotation is from Hamlet's famous "to be or not to be" soliloquy in Act III, Scene 1. By a "sea of troubles," Hamlet means life's many struggles. Hamlet is weighing the merits of life, which is, he argues, inherently full of travails and evils, against that of death, which would bring an end to these troubles:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune
Or to take arms against a sea of troubles,
And by opposing end them.
Is it better to endure the hardships of life, miserable as it may be, or "to take arms" against these troubles? By "take arms," Hamlet seems, in context, to mean committing suicide, and though he does not seem to be contemplating the actual act of killing himself at this point in the play, he is clearly disturbed, and is pondering whether life is worth living.
During Act I Scene 5, the ghost of Hamlet's father tells him to avenge his death and asks him over and over again to swear that he'll kill Claudius. Hamlet first struggles to determine if what the ghost says is real or not. If the murder happened as the ghost said it did, then surely Hamlet wants to seek revenge for his father's sake. Hamlet is not only confused and indecisive, but he's scared, too.
In Hamlet's famous soliloquy in Act III, he wrestles with the fact that going after Claudius could result in his death as well. By Hamlet questioning if he should "take arms against a sea of troubles" (III.i.61), it can be argued that he wonders if he should actually take up arms against his uncle who is now king. Once Hamlet takes up arms against his uncle in order to avenge his father, then he will open the proverbial can of worms. Hamlet wonders what other "troubles" could be the result of him deciding to wage war against his uncle. The worst case scenario is death--at which point Hamlet digresses and wonders if death is worth standing up to his uncle.
The line that follows "a sea of troubles" says "And, by opposing, end them" (III.i.62). Some may think this means that Hamlet is considering suicide, but if he is taking up arms against his uncle like he swore to do, then the troubles that would end would be the pressure to avenge his father's death. Either way, if Hamlet dies while fighting his uncle, then his troubles are over; but if Hamlet survives and Claudius dies, the sea of troubles are over with the fact that he finished the work that his father asked him to do.