There were two major ways in which the legacy of Rome was preserved by these empires.
First, the basic structure of Roman government was left more or less intact in places like the Ostrogothic Kingdom of Italy. In the Byzantine Empire, the Emperor Justinian codified Roman law, thus keeping it as a major source of Byzantine law. The Romans themselves were no longer ruling, but their basic governmental structure remained in use.
Second, all of these empires were Christian. Admittedly, many of them were not the same kind of Christian that the Romans had been. The Ostrogoths were Arians and the Byzantines were Orthodox. But it was still the case that Christianity of some form continued to exist. After some years, all of the kingdoms of Western Europe became what we now call Roman Catholic, further cementing the legacy of the Empire.