The text does not support this assertion. It is clear that in Act 1 scene 2, when Caliban is first introduced, he hates Prospero just as much as Prospero hates him. Although initially, when Caliban remembers what his relationship with Prospero had been like, it did seem that there might have been some affection, at least on his side, his subsequent enslavement and the way that he has been used and abused by Prospero clearly indicates that there is now no love lost between them. Note, for example, how Prospero calls to Caliban to come out of his cave:
Thou poisonous slave, got by the devil himself
Upon thy wicked dam, come forth!
This is hardly the language of somebody who holds the person being addressed in great esteem. In the same way, Caliban seems to use more insults in his speeches to Prospero than anything else, wishing a "southwest" wind to blow over Prospero and "blister" him. The text therefore strongly indicates that the relationship between Prospero and Caliban is only based on hatred and dislike. Prospero, before Caliban appears, says to Miranda that the only reason they keep Caliban is to "serve in offices / That profit us." Otherwise, it is inferred, they would do away with him in some way. There is therefore no ambiguity about their relationship, as it is based on hatred alone.
Prospero is somewhat justified in his enslavement morally, considering that Caliban tried to rape Miranda, which would have made her ineligible to marry Ferdinand later(although I don't know if he was that specific). I think their relationship has more hate in it than love.
Prospero does not view Caliban as a being who could be his equal, you could say that he is blinded by his prejudice against Caliban's appearance and manners. Because of this, it is easy for Prospero to justify enslaving Caliban. There was a lot of separation. Caliban's attempted rape, and his reaction when Prospero brings it up in the play futher the impression that he is more beast than human, and therefore can be used as a slave, rather than equal.
One theme of the Tempest is imperialism, and I think the treatment of Caliban is similar to the treatment that the inhabitants of the colonized islands received.