The details involving the loss of the dwarves' legendary treasure can be found in first chapter of The Hobbit, as Thorin retells the story to Bilbo.

As Thorin recalls it, the Lonely Mountain had been built up by his grandfather, Thror. Before this point, the dwarves had already fled from territories further in the north, resettling at the mountain and bringing all their wealth and possessions with them. At this point, the dwarves thrived. They expanded their halls and filled them with treasures so that the wealth of the dwarves (and of Thror, their king) became the subject of much renown.

The fame of their legendary treasure, however, drew the attention of the dragon Smaug. As Thorin tells Bilbo, dragons are rapaciously greedy. Unable to produce anything of value on their own, they instead steal their treasures in order to hoard them. In Thorin's account, there were many dragons living in the north, and thus the supply of gold and treasure was likely dwindling. Therefore, when Smaug learned of the treasure of the Lonely Mountain, the dragon relocated, attacking the dwarves and stealing the treasure for himself. The survivors of the attack fled into poverty, even as they would continue to retain the memories of what they had lost.