Here are a couple of ways to tell the difference: commercial fiction is often highly used for entertainment value and is sold often by the millions. These are often block-buster best seller books or books made into movies. I do not believe Jackson's Lottery fits this criteria of the definition. The John Grishams, Nicolas Sparks and Stephen Kings of today's era are commercial fiction.
Literary fiction is often studied in schools for the abilities to employ literary devices and contribute great purpose by the author. The Lottery certainly fits this mold. It's gothic horror, imagery, symbolism and evidence of the author's purpose fits the craftily constructed work that is classified as literary fiction.