The Dewey Decimal System is a system widely used by libraries, including school libraries, to catalog and shelve books. If you go to your own school library you are likely to see the Dewey letters and numbers on the spines of all the books. It has become very complicated over the years, as explained in the Wikipedia article which you can access by clicking on the reference link below this posting.
Although I have used the Dewey Decimal System all my life for locating the books I want in various libraries, I would have a hard time explaining exactly how it works, since I have never worked in a library. Each book the library receives is given a letter or sometimes two letters of the alphabet to indicate the type of subject matter the book contains. I have most frequently looked for books by English and American authors. The English books start with PR and the American books with PS. Then there are numbers apparently based on the year of publication, and there are usually decimals followed by more numbers to further categorize the individual books. It is actually very easy to find a book in a big library if they use the Dewey Decimal System and if you get the book's number off the computer before you start your quest. The books are all shelved alphabetically and then numerically in ascending order. You can read all about it in the Wikipedia article if you are sufficiently interested.
The book publishers seem to provide a Dewey Decimal classification code with each new book. For example, I have a hard-cover book containing both Animal Farm and 1984 by George Orwell. The copyright page (which is usually right on the back of the title page in any book) shows the Dewey classification as PR6029.R8A63 2003.
Harper Lee, who later criticizes public education with the words of Atticus in his closing remarks from Chapter 20, has Jem commit this malapropism about John Dewey, the philosopher, psychologist and educational reformer in order to indicate how little Dewey's progressive ideas have affected Jem.
John Dewey, a progressive thinker, emphasized democracy in all things, especially in the education of children. He felt that children should play an active role in their learning; therefore, the educator should not simply "dole" out facts and bits of wisdom. Instead, the educator should become a democratic "partner" in the learning process as she guides her students to discover meanings independently.
In Chapter 2 of To Kill a Mockingbird, Jem tells Scout,
I'm just trying to tell you the new way they're teachin' the first grade, stubborn. It's the Dewey Decimal System."
But, as is so well explained above, this system is one used to organize books in public libraries, also in grade and high school libraries. (But, not in colleges where the Library of Congress system is used, instead.)
Rather than saying the "Dewey Decimal System," Jem should have referred to John Dewey's system of education. In keeping with this, Miss Caroline has waved cards at the children on which are printed sight words such as the, cat, man, and you.