The distinction is basically artificial. Historically, the notion of the "figures" originated in Graeco-Roman education. This educational system included "grammar" as the main part of the secondary curriculum. Ancient grammatical studies consisted of explication of the poets (what we now would call literary studies) and the art of correct speech (what we now would term grammar and introductory composition). Rhetoric was one possible path of tertiary education, and focused on the art of advanced prose composition.

Figures of speech or "literary devices" were studied in both grammatical and rhetorical contexts, and thus overlapped literary and rhetorical contexts to a great degree. The main difference between the two is that while all the devices learned in the grammatical classroom were also used by rhetoricians, some of the persuasive strategies used in speeches were not introduced until the rhetorical level of schooling. However, the term "rhetorical devices", meaning such elements as climax, isocolon, anacolouthon, metaphor, or alliteration, is normally applied to figures of speech and thought which are used in both literary and rhetorical works. Such persuasive techniques unique to rhetoric as the enthymeme or the epicheireme are forms of argument, but not necessarily "devices" in the sense of figures of speech. 

Rhetorical devices are used to convey a particular meaning with the aim of persuasion or provoking an argument about a topic. These devices are mostly used in an argumentative or oratory environment were eloquence is necessary. Rhetorical devices are not strongly guided by grammatical mechanics but are mainly concerned with arrangement, style, delivery, memory and invention. These devices include sarcasm, metaphors and irony among other constructs. For instance a statement by Abraham Lincoln when he stated that his political rival "dived down deeper into the sea of knowledge and come up drier than any other man he knew” is well placed to show use of rhetorical devices. This statement uses both irony and a metaphor to persuade the audience that the rival did not benefit from education.

Literary devices are used to create the compression of ideas and uniqueness of expression that envelops story in metaphor, imagery and symbolism. Literary devices are attended to directly in the analysis, interpretation and appreciation of literature while they indirectly guide, generate and enhance (foreground) feelings, moods, tones, suspense and expectations for the reader of literature, which includes theatrical drama and poetry (in poetry, literary devices are called poetic devices).

Examples of literary devices include the literary element of plot, which is the sequence of events that determines a story, and the literary technique of flashback, which reveals plot in an a-chronological pattern and generates suspense. Mood is another literary element (also called atmosphere) that governs how a reader responds emotionally to a story and its setting (compare the mood of Poe's works with the mood of Austen's works). Another example of a literary device is the literary technique of oxymoron. For instance, in the phrase “terribly beautiful” the two words are used together to add emphasis but refer to each others' opposite meaning creating an oxymoron of contradictory meaning.

Authors use rhetorical devices mainly for persuasion. Rhetorical devices include repetition, emotional/logical appeal, irony, rhetorical question (a question that is meant to provoke thought, not an answer), parallelism (repetition of sentence structure), hyperbole (exaggeration), and compare/contrast. Rhetorical devices are important in political documents such as the Declaration of Independence and persuasive speeches.

Literary devices are also important in persuasion, but the list is a little broader. The purpose of a literary device is to bring meaning to a text and basically tell a story. There are dozens of literary devices, including alliteration, foreshadowing, flashback, allusion, metaphor, oxymoron, personification, and symbolism.

Rhetorical devices are more about the author's feeling toward a subject and his or her use of persuasion, while literary devices are more about telling a story.