Caliban despises Prospero. As far as Caliban is concerned, Prospero is a cruel master who torments and abuses him. When we first meet Caliban, he curses Prospero and hopes for "wicked dew" to "blister" him "all o'er." He accuses Prospero of taking the island from him, and then he curses Prospero again, wishing upon him the "red plague" and "all the charms of Sycorax." However, as much as Caliban despises Prospero, he acknowledges that he must, begrudgingly, do as he says, for Prospero's magic is "of such power."
When Caliban first meets Stephano, in act 2, he assumes that Stephano is an evil spirit sent by Prospero to torment him. Caliban is, therefore, initially very afraid of Stephano. However, later in the scene, Stephano gives alcohol to Caliban, and Caliban is so intoxicated with the alcohol that he thinks Stephano must be a god. He swears to be Stephano''s "true subject" and to "show (him) every fertile inch o'th' island." At the end of act 2, Caliban is ecstatic, believing that in finding a new master to serve he has freed himself from his previous master, Prospero. He shouts, "Freedom, hey-day! hey-day, freedom!"
Caliban is afraid of both Prospero and Stephano, because he believes that both possess magical powers that they can use to hurt him. We know, of course, that only Prospero has such powers. The key difference in Caliban's attitudes towards Prospero and Stephano is that he hates the former and feels grateful to the latter.