The brief answer to this is that Prospero needs to turn his attention to the plot that Caliban has hatched against him.  He knows that Caliban is plotting to kill him and he needs to take care of that problem.  As he says in the play

I had forgot that foul conspiracy Of the beast Caliban and his confederates Against my life. The minute of their plot Is almost come.—Well done. Avoid, no more!

That last line is spoken to the spirits, telling them that the play that they have been putting on was nice, but now it is time for them to leave.

This confuses Ferdinand and Miranda a bit because it seems somewhat abrupt--Prospero is enjoying this light play and then suddenly he orders the spirits to leave.