What have you, my good friends, deserv'd at the hands of
Fortune, that she sends you to prison hither?
Prison, my lord?
Denmark's a prison.
Then is the world one.
A goodly one, in which there are many confines, wards, and
dungeons, Denmark being one o' th' worst.
We think not so, my lord.
Hamlet:Hamlet Act 2, scene 2, 239–251
Why then 'tis none to you; for there is nothing either good or
bad, but thinking makes it so. To me it is a prison.
What brings Rosencrantz and Guildenstern—two of Hamlet's acquaintances from the university—to Denmark isn't Lady Fortune but, as Hamlet suspects, King Claudius. Claudius is worried about Hamlet's seeming distraction, thinking it might be a threat to the state and to the king himself. Claudius coerces Rosencrantz and Guildenstern, who aren't too bright, into service as spies, hoping they can lull the prince into revealing the true cause of his "antic disposition" [see ANTIC DISPOSITION].
When Hamlet calls Denmark a prison, therefore, the metaphor is apt. He is mentally and physically confined by the gaze of the king and his agents, and he feels trapped in the court's general degradation—"Something is rotten in the state of Denmark," as Marcellus had said [see SOMETHING IS ROTTEN IN THE STATE OF DENMARK].
Hamlet is a prisoner of his own thinking, and of his knowledge that his stepfather is a fratricide and his mother incestuous. When he states that "there is nothing either good or bad, but thinking makes it so," he's not indulging in ethical relativism as much as wishing for blissful ignorance. He's also implicitly damning the naïveté of the king's new yes-men.