Edgar Allan Poe is often called the father of the modern short story because of his own stories and also because of a famous passage he included in a review of Nathaniel Hawthorne's Twice-Told Tales published in Graham's Magazine in May 1842. This passage is worth serious study because it succinctly defines the modern short story. It has served as a guide to short story writers all over the world, and few if any have disagreed with Poe because it is so reasonable and so useful.
A skilful literary artist has constructed a tale. If wise, he had not fashioned his thoughts to accommodate his incidents; but having conceived, with deliberate care, a certain unique or single effect to be wrought out, he then invents such incidents, he then combines such events as may best aid him in establishing this preconceived effect. If his very initial sentence tend not to the outbringing of this effect, then he has failed in his first step. In the whole composition there should be no word written, of which the tendency, direct or indirect, is not to the one pre-established design. And by such means, with such care and skill, a picture is at length painted which leaves in the mind of him who contemplates it with a kindred art, a sense of the fullest satisfaction. The idea of the tale has been presented unblemished, because undisturbed; and this is an end unattainable by the novel. Undue brevity is just as exceptionable here as in the poem; but undue length is yet more to be avoided.
To put it in simpler terms, borrowing mainly from Poe: A short story is a dramatic narrative intended to be read at a single sitting and designed to produce a single effect. It is the "single effect" that distinguishes the modern short story, and as Poe says: "The idea of the tale has been presented unblemished, because undisturbed, and this is an end unattainable by the novel."
There are many different kinds of novels, but none can aim to produce a single effect--as for example the frisson produced by Shirley Jackson's "The Lottery" or by the great Edgar Allan Poe's "The Cask of Amontillado"--because novels are not intended to be read at a single sitting. Novels are customarily divided into chapters, and each chapter may produce its own effect. A short story should be read at a single sitting in order to produce the effect the author intends to produce; whereas a novel is not intended to be read at a single sitting and is usually divided into chapters because the considerate author will provide convenient stopping places. Some exceptionally long novels, such as Tolstoy's War and Peace, are divided into Books and the Books are divided into Chapters. Marcel Proust's masterpiece Remembrance of Things Past is divided into six or seven separate volumes.
The "single effect" of a short story does not have to be produced by a surprise ending, as in many stories by O. Henry, Maupassant, Saki, and many others. The effect can be produced by the overall mood of the story, as is often the case with Anton Chekhov. But in any case, the reader should be left with a single simple or complex feeling or impression or emotion which Poe chose to term a "single effect."
To define a novel is probably impossible, but if we know what a short story is, then we ought to be able to understand what a novel isn't.