Claudius is no fool. Although Polonius' explanation that Hamlet is suffering from a broken heart is plausible, he is not convinced. Even Gertrude thinks that Hamlet's madness is a result of his father's death and her hasty marriage. Perhaps Claudius is suffering from a guilty conscience and this fact makes him more suspicious that Hamlet knows more than he is letting on. He responds with "How may we try it further?" He wants to know for sure what the source of Hamlet's madness is. If it is caused by his relationship with Ophelia, Claudius can feel relieved.
This is why he and Polonius concoct the plan to use Ophelia as bait, so that they can spy on the Hamlet. They hope that Hamlet's interactions with Ophelia will provide them with the reasons that Hamlet is so melancholy. But Hamlet is too smart for either of them. He determines that he is being watched and refuses to play their game. He speaks harshly to Ophelia, but even more harshly to anyone who might be listening, threatening death to one married person (Claudius) and warning Polonius to play the fool in his own home. Claudius wisely concludes that "madness in great ones must not unwatched go."
Before Claudius and Polonius overhear the planned encounter between Ophelia and Hamlet, Polonius peaks Claudius' interest in Ophelia as the cause of his mad behaviour in Act II, scene ii. He produces a letter that Hamlet has written to Ophelia (and that Polonius has made her surrender to him) and reads it to Gertrude and Claudius. It is not Hamlet's love for Ophelia, Polonius claims, but her rejection of his love (insisted upon by Polonius) that has driven Hamlet to madness. He says:
And [Hamlet], repelled...
Fell into a sadness, then into a fast,
Thence to a watch, thence into a weakness,
Thence to a lightness, and, by this declension,
Into the madness wherein now he raves...
Claudius looks to Gertrude for her take on this explanation; she surmises that it is possible; and it is then, upon Polonius' suggestion, that they plan to set Hamlet up by placing Ophelia in his path to see what his reaction will be. And it is this "nunnery scene" in Act Three that convinces Claudius that spurned love is not the reason for Hamlet's madness.
In Act II of Shakespeare's Hamlet, Ophelia relates Hamlet's strange behavior, Claudius, who fears for his life and has already hired Rosencrantz and Guildenstern to question Hamlet, agrees to hide with Polonius and spy on the contrived meeting of Ophelia and Hamlet. When Hamlet meets with Ophelia, she feigns attendance upon maidenly matters. But, Hamlet is in a misogynistic mood and tells her he did not love her and she should go to a nunnery rather than "make monsters" of men.
Of interest are Hamlet's words about marriage which Claudius and Polonius overhear. Hamlet predicts that everyone who is presently married shall live except for one, and the rest shall stay as they are; that is, unwed:
God hath given you one face, and you make yourselves another. You jig and amble, and you lisp; you nickname God's creatures, and make your wantonness your ignorance. go to, I'll no more on't, it hath made me mad. I say we will have no more marriage. Those that are married already, all but one, shall live. The rest shall keep as they are. To a nunnery, go. (3.1.151-157)
After hearing these words, the fearful Claudius announces his plan to send Hamlet to England, ostensibly for Hamlet's safety, but in reality to distance him from Claudius. This scene is just another of those that illustrate the theme of Appearances vs. Reality