Legitimacy is generally defined as the idea that the government has the right to rule. A government that is deemed to be legitimate will be more likely to enjoy the support of the people because they will feel that it has the right to govern them.
One major source of legitimacy for the Roman government during the empire was the fact that people outside of Rome were allowed and encouraged to become Roman citizens of a certain level. This made them feel as if they were a part of the Roman polity and gave the Roman government more legitimacy in their eyes. That helped the Empire retain power over its citizens, thus contributing to its success.