The text does not support this assertion. It is clear that in Act 1 scene 2, when Caliban is first introduced, he hates Prospero just as much as Prospero hates him. Although initially, when Caliban remembers what his relationship with Prospero had been like, it did seem that there might have been some affection, at least on his side, his subsequent enslavement and the way that he has been used and abused by Prospero clearly indicates that there is now no love lost between them. Note, for example, how Prospero calls to Caliban to come out of his cave:
Thou poisonous slave, got by the devil himself
Upon thy wicked dam, come forth!
This is hardly the language of somebody who holds the person being addressed in great esteem. In the same way, Caliban seems to use more insults in his speeches to Prospero than anything else, wishing a "southwest" wind to blow over Prospero and "blister" him. The text therefore strongly indicates that the relationship between Prospero and Caliban is only based on hatred and dislike. Prospero, before Caliban appears, says to Miranda that the only reason they keep Caliban is to "serve in offices / That profit us." Otherwise, it is inferred, they would do away with him in some way. There is therefore no ambiguity about their relationship, as it is based on hatred alone.