Is Rosencrantz and Guildenstern Are Dead a tragedy or a comedy or something in between?
The element of tragedy is there but indirectly because it is based on a tragedy: Hamlet. This play is a comedy and a parody. Even when Rosencrantz and Guildenstern become serious and adopt the serious style more characteristic of Shakespeare's (and Hamlet's) language, this also sounds comical because it seems fake and a desperate attempt by Rosencrantz and Guildenstern to fit in with Shakespeare's characters. So, the play might seem like some hybrid of tragedy and comedy but since it is an outright parody of a serious play, it is actually more comedic than tragic.
Even the allusion to Samuel Beckett's Waiting for Godot, which is serious in being an homage, is humorous because it comes across like an inside joke. At the beginning of the play, Rosencrantz and Guildenstern are flipping coins, essentially passing the time or waiting (an obvious reference to Beckett's play). Rosencrantz claims that he's beaten a record and Guildenstern tells him, "Don't be absurd." This refers to the Absurd qualities of Beckett's play and is therefore, a reference to the silliness of an absurd situation and an inside joke (by referring to Waiting for Godot).
Rosencrantz and Guildenstern talk about serious things (i.e. death and time) but they do so in a comical and playful way. In Act II, Rosencrantz even says that it's silly to worry about death:
I mean one thinks of it like being alive in a box, one keeps forgetting to take into account the fact that one is dead . . . which should make a difference . . . shouldn't it? I mean you'd never know you were in a box, would you? It would be just like being asleep in a box.
The use of self-reflexivity is also used to comic ends. For example, when the Players perform The Murder of Gonzago, Guildenstern asks who decides who lives and dies in the "play within a play." The Player says that "It is written." Thus, The Murder of Gonzago is a play within a play (Hamlet) within a play (Rosencrantz and Guildenstern are Dead). This blurs the line between art and life. Guildenstern later says he'd prefer art to mirror life. But as the lines are blurred, life could mirror art. This suggests the absurd silliness that Rosencrantz and Guildenstern think they are real but have been given clues that they are stuck in a play themselves.