Caliban is the son of Sycorax, a witch who had controlled the island before Prospero got there. We are not exactly told how Prospero got control of the island, but since he landed there, he has been the one in control of the island.
Prospero is able to control the spirits of the island. By using these spirits, he is able to control Caliban as well. If you look at the parts where Caliban talks to Stephano, you can see him talking about how Prospero uses spirits to control him -- he talks about how they will hurt him if he does not obey Prospero.
Prospero has clearly got some control over Caliban. However the extent of control is substantially lower than you would have thought. In act 1 scene 2, Miranda and Prospero visit Caliban. Miranda obviously doesn’t care about control over him, “I do not love to look on,” all she wants is too stay away from him. This is not her only vain moment in the play. Prospero however makes a great show of how much power he has, how he can punish Caliban, yet Caliban is completely used to this form of abusive behaviour that he replies with a non-sequitur. He is obviously immune to his punishments by now. That shows that he just ignores Prospero most of the time, and doesn’t see him as a threat. However Prospero can control some aspects of Calibans life. He controls where Caliban lives, what Caliban does, and where he is allowed to go. Prospero cannot control what Caliban says to him. Prospero taught him their language, so Caliban must have heard it from Prospero first. In conclusion, Prospero controls Caliban in the only way he knows how, with magic.