This is a very important of the play, as Prospero declares his intention to renunciate his magical powers as he has achieved the objective he has been aiming for. However, there is a sense in which Prospero seems to recognise that the magical powers he has used to successfully on the island are only suitable to be used there. His return to civilised life indicates that he must leave this period of life behind him. The magical island as a setting means that it is suitable for him as a character to practice magic, but the court setting from which he was banished and will be returning to is a place of politic and intrigue, and therefore magic is not appropriate. Thus the magic that Prospero practises is "rough," because it is not suitable for the sophisticated court life that Prospero will be returning to.
Likewise, although Prospero has achieved so much with his magic, perhaps the description of his magic as "rough" recognises its limitations. He could control the elements, creating storms and shipwrecking vessels, but he was not able to change emotions and ways of thinking. He could not change the hearts of Antonio or Sebastian for example. Thus, his magic is at best a "rough" instrument, unsuited for the delicate task of changing people's thoughts and attitudes.