What is the definition of a tragedy in literary terms?

The first great theorist of dramatic art was Aristotle (384-322 B.C.) whose discussion of tragedy in Poetics has set the bar for what constitutes tragedy.

• According to Aristotle, tragedy is the imitation in dramatic form of an action that is serious and complete, with incidents arousing pity and fear wherewith it effects a catharsis of such emotions (an emotional release) at the end. 
• The best tragic plots involve a reversal of fortune or a discovery (change from ignorance to knowledge) or both.

Here is a description of the tragic protagonist:

1. The tragic hero is a man/woman of noble stature.
2. The tragic hero is good, though not perfect, and his/her fall results from his/her committing what Aristotle calls "an act of injustice" (hamartia) either through ignorance or from a conviction that some greater good will be served.  The protagonist is personally responsible for his/her downfall.
3. The hero's downfall is tragic rather than merely pathetic. 
4. Nevertheless, the hero's misfortune is not wholly deserved.  He/She is a person mainly admirable, and the fall, therefore, fills the audience with pity and fear (catharsis).
5. The tragic fall is not pure loss.  He/She exits accepting his/her fate and acknowledging that it is to some degree just.
6. Tragedy does not leave its audience in a state of depression.

It is important to note that there is a critical tradition that attributes the fall of the hero to a "tragic flaw"--some fault of character such as inordinate ambition, quickness to anger, a tendency to jealousy, or overweening pride.  This definition seems more appropriate to "Macbeth" as you have tagged your question as his inordinate ambition (along with Lady Macbeth's influence) is the driving force for his downfall.

What is the definition of a tragedy in literature?

In literature, a Tragedy is a story which "treats in a serious and dignified style the sorrowful or terrible events encountered or caused by a heroic individual" (Encyclopedia Britannica) and shows the universal role of Man within these events.

The tragedy is a common and popular form of storytelling, with examples stretching back thousands of years. The first instances of tragedies occurred in Ancient Greece, around the 5th Century BC, and were complex song-and-dance plays performed to honor the god Dionysos. The term "Greek Tragedy" is still used today. The tragedy evolved over the course of the years; Aristotle defined a tragedy as:

...an enactment of a deed that is important and complete, and of [a certain] magnitude... and through pity and fear [for the hero] it effects relief to such emotions.
(Aristotle, Poetics, VI 1449b 2-3)

The tragedy today rejects the upper-class mentality (kings, nobility) that permeates most classical tragedy to focus on the ordinary citizen, who is often beaten down by society, larger events, and the knowledge that one may be an unimportant cog in the greater scheme of things. Arthur Miller's Death of a Salesman is an example of a modern tragedy.

One of the best-known examples of a tragedy is Shakespeare's play Hamlet, (full title: The Tragedy of Hamlet, Prince of Denmark) which concerns the doomed Prince of Denmark  and the various schemes and events which lead him and many others to their deaths. The play is thematically and historically dense, filled with allegory and symbolism, and shows the necessity of Man to understand or accept mortality while still striving to live. Hamlet's famous soliloquy demonstrates these themes:

But that the dread of something after death
The undiscover'd country, from whose bourn
No traveller returns, puzzles the will,
And makes us rather bear those ills we have
Than fly to others that we know not of?
(Shakespeare, Hamlet, eNotes eText)