The word is not "mistrel" but "minstrel." Minstrels were traditionally wandering musicians who played stringed instruments and sang songs, often about love and brave deeds. A minstrel who was carried away with emotion when singing might be said to be experiencing rapture, which I take to mean a powerful and pleasurable emotion. Such a strong emotion could easily cause the minstrel to sing in a stronger, louder, higher or deeper tone, which could be described as swelling. The line you quote could be interpreted to mean that no minstrel would sing about the person at all or else that he would not sign with much feeling. Probably Scott meant that no minstrel would ever sing about such an individual at all because that person had never done anything noteworthy. No doubt the medieval minstrels were usually singing with rapture because it was expected of them, so saying that no raptures swell would be the same as saying that no songs would ever be sung about the wretch concentered all in self.