Tired as he was, Bilbo would have liked to stay awhile. Elvish singing is not a thing to miss, in June under the stars, not if you care for such things. Also he would have liked to have a few private words with these people that seemed to know his name and all about him, although he had never been with them before.
So, Bilbo wanted to stay because he wanted to enjoy the elves' singing, and because he wanted to know more about them. Recall that Bilbo had never had any adventures or been very outside of his home, and had rarely met an elf, and the great majority of a hobbit's knowledge of things outside the Shire was anecdotal or outdated, although much of this information comes to light in greater detail in the Lord of the Rings, and is simply implied in The Hobbit. For Bilbo, meeting an elf, let alone visiting one of their major strongholds, was part of the appeal of the journey, inspired at least in part by his un-hobbit-like curiosity.
This passage comes shortly after the elves' introductory song, with a short description of their nature and relationship with dwarves, just before the party continues onward to the Last Homely House (Rivendell) and consult with Elrond regarding the moon-runes on the map.