This is the most difficult question I have ever attempted to answer on this site and perhaps off of it as well. Many literary scholars say that one should not even make the attempt. Lord David Cecil remarked that he could not prove to anyone that Ian Fleming, author of the James Bond books, was not the greatest writer who ever lived. Certainly nothing approaching mathematical proof is possible. There is an iconic scene in the film Dead Poets Society in which the new teacher, Mr. Keating, played by Robin Williams, asks a student to read out a section from the introduction to a poetry anthology, in which the editor explains how to plot a graph measuring the excellence of a poem. He then instructs all the students to rip out the introduction, since it is an exercise in philistinism.
John Carey, former Merton Professor of English Literature at Oxford, has argued that artistic judgment is entirely subjective. It is for you to judge whether a book or poem has any merit for you personally. This does not take account of the astonishing level of consensus that Shakespeare, for instance, is a great writer. There will undoubtedly be a strong element of subjectivity determining which writers you prefer. This will include not only how much you enjoyed a book or a poem, but how often you think about it afterwards. I would suggest that this is why Lord David Cecil singled out Ian Fleming’s books as an example of popular literature which, although he could not prove its inferiority to Shakespeare's, he regarded as unworthy of serious study. I personally have read all the James Bond books and enjoyed them. However, I do not think about them very often—perhaps a few times a year—whereas I think about Shakespeare several times a day. It seems to me that one of the most important measures of quality in a book is not only how much you enjoy it at the time, but how much you think about it afterwards, which is as good a measure as any of the effect it has had on you.
All the possible measurements of goodness or badness in a book are subjective to some extent. Some are very subjective indeed. Here are a few suggestions, beginning with those I have already mentioned, on how to make a meaningful distinction.
A good book is a book you enjoy. A bad book is a chore.
A good book is one you remember and think about afterwards. A bad book is quickly forgotten.
A good book contains believable, relatable characters. A bad book contains caricatures and cardboard cutouts.
A good book deals with compelling, perhaps universal themes. A bad book is trivial, boring, or both.
A good book uses language skillfully. It contains striking, memorable images and phrases that stick in you mind. A bad book is monotonous.
A good book influences you, even changes you. A bad book has no effect.
Even these very general thoughts do not apply all the time. Charles Dickens is a great novelist, despite the fact that his characters are generally caricatures. Marcel Proust is a great novelist who has long passages of great detail which bore many readers (Somerset Maugham said he would rather be bored by Proust than entertained by anyone else). As with any of the arts, extensive experience refines the taste and allows the reader to appreciate a wider range of literature. Still, I believe that if you were to make a note of every time an author came to mind for, say, a month, you would find that the books sort themselves out into an order of quality and importance by the effect they have had on your mind.