Claudius knows that Hamlet was the legitimate heir to the throne, and that he himself is a usurper; and he is afraid that Hamlet might either lead a coup against him or that some foreign enemy might persuade Hamlet to act as a figurehead in a plot to overthrow the existing government under King Claudius and set up a puppet government with Hamlet as the new king. Hamlet, according to Claudius, "is loved of the distracted multitude."
Claudius doesn't understand Hamlet. He wishes he could. He summons Rosencrantz and Guildenstern to the court specifically in order to get them to spy on Hamlet and report to him. He encourages Polonius to spy on him. He feels that Hamlet is a ticking timebomb. In one of Shakespeare's most beautiful metaphors, Claudius says of Hamlet:
There's something in his soul,
O'er which his melancholy sits on brood,
And I do doubt the hatch and the disclose
Will be some danger . . .
Claudius throughout much of the play is in a awkward position because he doesn't feel he can act directly against his moody stepson, either by having him killed or imprisoned. He probably is content to let Hamlet inherit the throne after his own demise, but he wants to retain power for a long time. (If Gertrude were to have a child, that would probably be a different story--and maybe that is why Hamlet is so upset by all the lovemaking that goes on in public and private between his mother and his lusty stepfather.)
Claudius explains at one point that he can't harm Hamlet because Gertrude loves her son so strongly that "she lives by his looks" and he loves Gertrude just as strongly. Otherwise he would probably dispose of Hamlet the same way he disposed of Hamlet's father. Hamlet knows his life is in constant danger. This is the main reason he acts insane. He wants to be thought of as a harmless lunatic, even a fool. He does this because he knows he is under suspicion or who might be spying on him.
Claudius's policy before the death of Polonius is to keep Hamlet under close scrutiny and isolated from anyone who might conspire with him to stage a coup or a revolt. The last thing Claudius wants to do is to allow Hamlet to leave the country, where he would have opportunities to participate in an armed invasion of Denmark, similar to that staged by Malcolm at the head of an English army in Macbeth.
Claudius doesn't really explain why he wants Hamlet to remain at court instead of returning to Wittenburg, where he has been studying. He simply tells Hamlet, after expressing his hopes that he will view him as a father, that his return to Wittenberg "is most retrograde to our desires." Essentially, he says that he just wants the young Hamlet around, "in the cheer and comfort of our eye," and he reminds Hamlet that he is the "chiefest courtier, cousin, and our son." Later, of course, Claudius wants him gone, particularly after he stages the play. He remarks "how dangerous is it that this man goes loose!" after Hamlet has killed Polonius, and, of course, attempts to send him the England to his execution. It is possible that Claudius wanted to keep Hamlet nearby because he didn't trust him, but it would have made perfect sense for Claudius to ensure that he had the allegiance of the dead king's son under the circumstances.