While a case could be made that literature is not a matter for children, it seems to me that the phrase "children's literature" is not at all an oxymoron.
An oxymoron is the use of two contradictory terms combined to create a kind of foolish paradox. In essence, the two words are almost mutually exclusive. A common oxymoron is "jumbo shrimp," and we know that such a thing exists, despite the usual meaning of the two words.
In the case of children's literature, the "foolishness" is suggested by the fact that good writing that stands the test of time because of its themes and other elements has a weightiness which should not be paired with the word "children's."
For example, children are not capable of understanding the sin and shame associated with Hester Prynne's sin of adultery (as in The Scarlet Letter), nor are the tragic elements of Oedipus Rex (killing his father, sleeping with his mother, his relationship with his children/siblings) understandable by children. Both of these are probably true; however, the issue is bigger than the fact that some literature is not suitable for children or their limited understanding. Literature is possible at a child's level.
Consider these definitions of literature:
written works (such as poems, plays, and novels) that are considered to be very good and to have lasting importance
writings in which expressions and form, in connection with ideas of permanent and universal interest, are characteristic or essential features, as poetry, novels, history, biography, and essays.
No mention of "adult" themes or audiences. Good writing is timeless and universal, and it can be written for any audience, including children.
While it is true that much of what is most beloved about children's literature is often based on memorable characters. it is possible to place those characters in novels, poems, or short stories that speak universal truths. One excellent example of this is The Lion, the Witch and the Wardrobe by C.S. Lewis. While we enjoy memorable characters like Mr. Tumnus, the White Witch, and Aslan, we are also prompted to think about our own sins and guilt as we follow Edmund's journey. We are moved by the sacrifice, forgiveness, and joy which we find in Aslan and appalled by the machinations of evil until they are atoned for by something greater than hate: love.
These are the kinds of things that stand the test of time and represent something beyond memorable characters, amazing drawings, or catchy verses. Those are not the elements which determine whether a work is deemed literature. Timeless literary works, written for children or adults, has value to readers of every age. Children's literature is not less than, it is more than because it appeals to all of us.