When Prospero and his daughter, Miranda, first come ashore on their remote island in Shakespeare's The Tempest, the only inhabitants of the island are a spirit, Ariel, and Caliban, whom Prospero describes as "A freckled whelp hag-born—not honour'd with / A human shape." (1.2.334–335).
Ariel was the slave of a witch named Sycorax, who was Caliban's mother. Prospero freed Ariel from inside a pine tree where Sycorax had imprisoned him, and where he had been trapped for twelve years after Sycorax's death.
Prospero takes Ariel as his own slave, and when Ariel complains about his enslavement or his duties, Prospero threatens to imprison him in an oak tree for another twelve years.
PROSPERO. If thou more murmur'st, I will rend an oak,
And peg thee in his knotty entrails till
Thou hast howled away twelve winters. (1.2.347–349)
Ariel's fear of imprisonment and Prospero's promise to one day free Ariel from slavery is how Prospero controls Ariel.
As for Caliban, Prospero felt sorry for him and took him into his home.
CALIBAN. ... When thou cam'st first,
Thou strok'st me and made much of me, wouldst give me
Water and berries in't, and teach me how
To name the bigger light, and how the less,
That burn by day and night; and then I loved thee,
And showed thee all the qualities o'th‘ isle,
The fresh springs, brine-pits, barren place and fertile—
Cursed be I that did so! All the charms
Of Sycorax, toads, beetles, bats, light on you!
For I am all the subjects that you have,
Which first was mine own king, and here you sty me
In this hard rock, whiles you do keep from me
The rest o‘th’ island. (1.2.395–407)
Prospero now keeps Caliban as a slave and confines him to a rock on the island for taking advantage of Prospero's kindness.
PROSPERO. ...I have used thee,
Filth as thou art, with human care, and lodged thee
In mine own cell, till thou didst seek to violate
The honour of my child. (1.2.409–413)
Miranda, too, has something to say about Caliban's behavior.
MIRANDA. ...I pitied thee,
Took pains to make thee speak, taught thee each hour
One thing or other. When thou didst not, savage,
Know thine own meaning, but wouldst gabble like
A thing most brutish, I endowed thy purposes
With words that made them known. But thy vile race,
Though thou didst learn, had that in't which good natures
Could not abide to be with; therefore wast thou
Deservedly confined into this rock,
Who hadst deserved more than a prison. (1.2.419–428)
Caliban is unrepentant and continues to be hateful toward Prospero and Miranda.
Prospero controls Caliban by keeping him isolated on a rock on the island and by threatening him with physical pain.
In his first appearance in the play, Caliban curses Prospero, and Prospero immediately threatens him.
CALIBAN. As wicked dew as e'er my mother brushed
With raven's feather from unwholesome fen
Drop on you both! A south-west blow on ye
And blister you all o'er!
PROSPERO. For this be sure tonight thou shalt have cramps,
Side-stiches that shall pen thy breath up. Urchins
Shall, for that vast of night that they may work,
All exercise on thee. Thou shalt be pinched
As thick as honeycomb, each pinch more stinging
Than bees that made ’em. (1.2.383–392)
Later in the scene, Prospero orders Caliban to fetch firewood, and Caliban ignores him. Prospero threatens Caliban again.
PROSPERO. Hag-seed, hence!
Fetch us in fuel, and be quick, thou'rt best,
To answer other business.—Shrug'st thou, malice?
If thou neglect'st or dost unwillingly
What I command, I'll rack thee with old cramps,
Fill all thy bones with aches, make thee roar,
That beasts shall tremble at thy din. (1.2.432–438)
Caliban is quick to obey in response to the threat, and remarks about the force of Prospero's magic.
CALIBAN. No, pray thee.
[Aside] I must obey. His art is of such power,
It would control my dam's god Setebos,
And make a vassal of him. (1.2.439–442)
Although fear can be an effective way to control someone—and is an effective way for Prospero to control Caliban, who appears to responds only to threats of physical violence and pain—fear can also provoke hatred and resentment toward the person who causes that fear.
Caliban's hatred and resentment toward Prospero is manifested in Caliban joining Stephano (who Caliban thinks is a god) in Stephano's plot to overthrow and kill Prospero.
With that in mind, Prospero's threats and his attempts to control Caliban by fear, although effective, are entirely counterproductive and could have resulted in Prospero's death.