Shakespeare inherited his concept of tragedy from ancient Greece and Rome, as well as contemporary writers that wrote just before him or simultaneously with him. Tragedy should involve a tragic figure with great power. The figure must possess enough power to do great harm. The average individual doesn't have that kind of power, so a member of royalty usually serves as the tragic figure.
This tragic figure must possess a tragic flaw that leads to his downfall. Ambition, for example, is Macbeth's flaw. Hamlet's inability to act is his. If Hamlet were more single-minded and had achieved his revenge as soon as he was sure Claudius was guilty, the tragedy at the conclusion of the play could have been avoided.
Macbeth is a more traditional tragedy, more classical. Hamlet, however, transcends the traditional. Shakespeare takes what he inherited from others and goes beyond it in Hamlet by creating a tragic character who is extremely complex.
Finally, the tragedy should create a cleansing, a catharsis, not only in the play's setting (Denmark in Hamlet), but in the audience.