The revenge play was a popular genre during Shakespeare's time, one that audiences found quite entertaining. Hamlet features the many standard conventions of the revenge play, but in Shakespeare's hands it became an enduring literary masterpiece. The difference between Hamlet and the revenge plays of Shakespeare's time is found really in your question--the word "tragedy." Revenge plays were not tragedies, but Hamlet certainly is, the tragedy developing from one of the most complex characters in literature, the young Prince of Denmark.
It has been said that Hamlet is, in fact, the first introspective character in English literature. The power of the drama develops not from complicated external plot events but from Hamlet's tortured examination of his own character--his feelings, his vices, his genuine and perhaps unrecognized motivations. As the basic revenge plot advances, Hamlet is drawn more deeply into himself, often reacting with confusion, guilt, and even self-hatred at what he discovers. Unlike other revenge play protagonists, Hamlet shrinks from what he feels compelled to do; his search for moral certainty traps him in inaction and further self-contempt. Because of his very nature, he becomes a psychological prisoner of circumstances he did not create and cannot change. It is the complexity of Hamlet's character and his tortured introspection that lead to his destruction and elevate Shakespeare's drama into the realm of unforgettable tragedy.