The main connection between nationalism and Gaddafi’s attempts to legitimize his rule is that nationalism impeded those attempts. Gaddafi had a hard time legitimizing his rule due to both Libyan nationalism and Islamic nationalism.
On the religious front, Gaddafi could not really present himself as an Islamist or Islamic fundamentalist. He was fairly secular and had begun his career as a Nasser-style Arab nationalism and socialism. This meant that he could not appeal to Islam as a way to legitimize his rule.
With regard to Libyan nationalism, Gaddafi also had problems. He had been more of a pan-Africanist and pan-Arabist for most of his career. He had not really legitimized his rule by trying to create a sense of Libyan patriotism. This may be partly because of the kinds of ethnic and tribal splits that exist in the country. At any rate, Gaddafi relied largely on foreign mercenaries for his security. Many of the pro-Gaddafi forces at the end of the rebellion were mercenaries from other African countries such as Niger and Mali.
For these reasons, nationalism had a negative impact on Gaddafi’s attempts to legitimize his rule.