I believe it is possible to argue either way; however, if I had to pick one or the other, I would say that "The Lottery" should be considered literary fiction. Generally speaking, commercial fiction is faster paced than literary fiction. The author will focus on building a strong plot in terms of events and conflicts. Characterization isn't usually central to the story. In contrast, literary fiction is much less frantic in pace. Style, characterization, and complex themes are par for the course. A decent litmus test between the two formats is to ask yourself about the ending. Commercial fiction tends to end happily. Literary fiction rarely ends happily. There are going to be exceptions to that rule, but it tends to work more often than not. It's also possible that something intended for commercial purposes winds up becoming a source for literary study. The opposite can become true, too, so the categories themselves do allow for some dynamic fluidity. If you use my happy ending litmus test, then "The Lottery" clearly fits itself into literary fiction. It ends with a woman being stoned to death for no other reason than tradition. The story begs discussion of the complex themes of tradition and change. "The Lottery" also isn't deep in terms of plot. Events are not driving the story. People and their actions and attitudes are driving the story.