Remember, when Hamlet refers to Denmark as a prison, he is speaking to Rosenkrantz and Guildernstern:
What have you, my good
friends, deserved at the hands of Fortune that she sends you
to prison hither?
He knows that the pair have been sent for and that they are spying on him. He also knows that whatever he says will go back to his murderous uncle, King Claudius. Now also remember why he is still living in the castle and not back at school: Claudius, the King, has requested that he remain there, under his watchful eye. He is a prisoner of the King and he wants the King to know he feels that way.
And further, as noted above, he is a prisoner of unpleasant and pressing circumstances that have, much to his displeasure, fallen upon him and constricted his accustomed freedom of motion and action. He's caught in Denmark by decree and circumstance. A prison, indeed.
In Act 2, Scene ii Hamlet refers to Denmark as a prison because he feels so trapped in his life there and feels helpless to change his situation, as if he were locked into it like a prison cell.
In the case of Hamlet, any of the control he thought he had, fell away with the murder of his father. Having his father, the king, be killed by his own brother has sent Hamlet slightly mad. This feeling of helplessness and feeling out of control is exacerbated by the fact that he is cooped up in a palace with no real outlet.