A literary convention is:

A customary feature of a literary work, such as the use of a chorus in Greek tragedy, the inclusion of an explicit moral in a fable, or the use of a particular rhyme scheme in a villanelle. Literary conventions are defining features of particular literary genres, such as the novel, short story, ballad, sonnet, and play.

In other words, it is a cliche, device, or trope that acts as a defining feature of a genre. All Star Wars movies begin with the phrase "A long time ago in a galaxy far, far away." This helps place the viewer in the context of the Star Wars galaxy. Literary conventions can be aspects of prosody (rhyme and sound), structure (acts in a play), or content (humor in a comedy). 

To elaborate on a few of the examples provided above:

• Chorus - In Greek tragedy, the chorus is a group of masked performers who provide context for the events that unfold. They do not directly interact with the action performed by the actors but can provide prologues, epilogues, and commentary for the benefit of the audience. 
• Moral - Fables are designed to communicate a moral lesson about how the audience should or should not act. This moral is often communicated in the form of a single line or utterance that summarizes the lesson of the fable. An additional convention found within fables is that the actors in the story are not human.
• Rhyme Scheme - Poems often use rhyme as a convention to distinguish themselves from prose. Such poems follow a pattern of rhyming sounds—the first and third lines, for example, must rhyme, while the second and fourth do not. Poetry does not require rhyme, however, and utilizes many different conventions based on style, language, and purpose. 

A literary convention is a commonly used device or aspect of a certain kind of work. If you are reading a certain type of literary work, you can expect it to conform to certain conventions. For example, plays are typically divided into acts and scenes, and characters generally speak to each other with dialogue. In Shakespeare's dramas, characters communicate their inner thoughts through the use of soliloquies. If you do not understand this particular literary convention, it will be difficult for you to make sense of what is going on--you might think that the character is speaking to other characters rather than privately expressing their inner thoughts. Some literary forms, like the novel, for example, have fewer conventions. (For example, some novels are broken up into chapters, and some are not.)