There are a number of ways that literature is classified having to do with purpose, content, style, and form. The most basic categories in literature are fiction and non-fiction.
These are divided according to a work's purpose and content in the sense that the purpose of non-fiction is to tell a true story (purpose) that is constituted of actual facts and/or honest opinion (content). Fiction does not have to be true, actual, factual or honest in the opinions expressed by the author.
Since fiction is imaginary, any novel, such as those of Austen, Brontë, Cather, or Conrad, may be included in this genre, in addition to any other “made-up” work of literature.
Within non-fiction there are multiple major genres: biography, autobiography, memoir, creative non-fiction, journalism, and informative works like text books.
Within fiction there are many genres, some of which overlap. Some of the best known genres are mystery, western, detective/noir, science fiction, fantasy.
Also, "pulp fiction" and literary fiction are two categories that help to identify works in terms of the writer's intentions (to entertain or to produce art, respectively). These categories, again, are not rigid. A work can be both a piece of fantasy and a western and a work of pulp fiction at the same time.
While the term genre does not typically describe the difference between fiction and non-fiction, but rather distinguishes between books within these categories, this the term that best answers your question here.
Genre - a category or class of artistic endeavor having a particular form, technique, style, or content. Some current genres are the novel, short story, essay, epic, tragedy, comedy, satire, and lyric.
Additionally, books that strive to prove a point or make an argument can be called polemical books or polemics, whether fiction or non-fiction.
Another major genre of literature is drama, which has its own sub-categories.