What exactly is the definition of an anti-hero in literature?
One definition of an "anti-hero" in literature is:
A central character in a work of literature [that] lacks traditional heroic qualities such as courage, physical prowess, and fortitude.
The perfect example of an anti-hero comes to mind: Grendel from the epic tale, Beowulf. Grendel has physical prowess, but rather than courage, he is driven by his hate and blood lust.
He makes himself the "captain of evil" by hunting [the inhabitants] at Heorot and devouring them with demonic vigor.
Fortitude is defined as "mental and emotional strength in facing difficulty:" Grendel has neither of these. Essentially, he is a bully. When he first realizes that he has met his match in the person of Beowulf, his first desire is to flee. When he is fatally injured, his only thought is to return to the fens to his mother, who he generally treats with total apathy and disregard, feeling himself far superior to her.
The definition goes on to note:
[Anti-heroes] usually accept, and often celebrate, their positions as social outcasts.
Grendel is a character of evil that is described as a descendant of Cain, son of Adam and Eve, who killed his brother Abel. Beowulf recounts that for this, Cain's descendants were cursed, and Grendel is one of those descendants. Beowulf also tells the story of one brother killing another:
So Grendel carries the aura of being the ultimate outcast.
Grendel is described as a creature that resents the joy, music and camaraderie of men as they gather in the mead hall, but rather than causing him to hide and wallow in misery, he takes pleasure in destroying all those that visit the hall.
Grendel also lacks courage: instead of attacking humans during the daylight hours, he comes "as a thief in the night." He has placed upon himself a spell that prevents the weapons of man from harming him. "Ancient Nordic society" would have seen these things to...
...contraven[e] the rules of fair play in combat.
Grendel, a central character in Beowulf, is a cowardly outcast that does not embody the characteristics of a hero.