Keep in mind that the McCourts literally have nothing--no material possessions. One could even argue that the children did not even have the love of their father seeing as how he chose alcohol over feeding for and caring for them on a daily basis.
The songs and stories in this book are Frankie's way of having SOMETHING to hold on to, if only in the figurative sense. Those stories and songs represent private, quiet, one-on-one moments that Frankie was able to share with his father, moments that were few and far between. They were truly the only thing that his father ever gave to him, and to have to share that would have been horrible.
In a family where even the smallest and most meager items of food, clothing and space are shared, these stories are the one thing that Frankie has all to himself; they represent the love of his father.