Denmark is a prison for Hamlet because he has realized that even his childhood friends cannot be trusted and that he is being spied on by many. Everyone seems to be watching him, much like prisoners are watched in a prison. In fact, Hamlet says that the world is a prison and Denmark is one of the worst dungeons because he feels there is no one he can trust. He feels betrayed by everyone, even himself at this point because the actor in the play was more intense in his revenge than Hamlet has been in his own real-life plot to avenge his own father's death. He feels trapped and alienated at this point in the play- feelings which only intensify as the plot unfolds.
When Hamlet asks Rosencrantz and Guildenstern why they would volunteer to come to a prison, they respond that they don't think of Denmark as a prison. Hamlet explains, “[T]here is / nothing good or bad but thinking makes it / so.” Hamlet realizes that Denmark seems a prison to him alone due to his personal experiences there since his father died. In that case, let's compare an actual prison to Hamlet's recent experiences in Denmark.
A prison is run by a warden, who will not let you leave or even make your own decisions until your term is up. Claudius has taken the throne and will not allow Hamlet to return to university in Wittenberg, telling him to get over his “unmanly grief” concerning his father’s death. Ironically, what a young man of Hamlet’s intelligence and spirit needs most for healing is to break out of the confines of Denmark and go back to school. Hamlet is not even given a time limit to his term here in Denmark.
In prison, one's behavior often becomes more immoral than it was on the “outside.” We certainly see Hamlet's moral and mental character degrading as he attempts to deal with the oppression he is put under by the demands of a murdered father; the expectations of Claudius, Gertrude, and their court; the sorrowful questioning of the woman he once loved; and the insulting behavior of everyone spying on him. No wonder he chooses to “put an antic disposition on.” The desire to run and hide from it all, to gain a moment of freedom, must be overwhelming.
In this scene with his former childhood friends, Hamlet feels most imprisoned by the constant spying that he is subject to. Claudius and his sidekick Polonius are hiding behind every curtain, setting him up and expecting him to fall for it. He can’t even call them out on it for fear of exposing his knowledge that Claudius is a murderer, putting his own life in danger. His mother sides with her new husband/former brother-in-law at every turn. (Another irony—the warden of the prison is more immoral than the prisoner.) Claudius turns Hamlet’s friends and loved ones into spies at the snap of his royal finger. Even Ophelia, with whom Hamlet seemed deeply in love prior to all this upheaval, joins in the falsehood, helping Claudius spy on him. But truth be told, Hamlet would not have likely been allowed to marry her anyway. As prince (sounds a bit like the start of “prisoner,” doesn’t it?) he would be ordered to marry a woman of royalty.
All things considered, as heir to the throne, Prince Hamlet has probably not had a whole lot of freedom his whole life, and dealt with it just fine. It is not until a new, crooked warden takes over that Hamlet begins to shake his prison bars.