A tragedy is defined as a story that treats a sorrowful subject in a dignified way. A tragic hero is a character that is brought down by his/her own flaws.
The story of Hamlet deals with the mortality of humans, the issues revolving vengenance, and the complexity in relationships. All serious and often sorrowful subjects. Hamlet is plagued by an inability to act, and gets trapped into situations as a result of it. His conflicts with his mother, Claudius and Ophelia all result from his inability to act on his impulses. Polonius' death, Ophelia's death, and the plot to kill Hamlet all stem from this flaw.
However, Hamlet recognizes his flaw. He stands up for himself and makes a conscious choice to return to Denmark and fight. He does this early - unlike in most traditional tragedies, the realization is the climatic moment as opposed to the resolution. And it is his action, his return to Denmark, that actually brings about the tragic ending. Shakespeare defies tradition by having his hero fail after he overcomes his flaw. This helps to demonstrate his theme of mortality, to show that all humans - hero and villian, rich and poor - are subject to the whims of the universe and to a finite life.