The "play-within-a-play" is this acting out of "The Murder of Gonzago," or, has Hamlet has renamed it, "The Mousetrap," inside the play by Shakespeare, Hamlet.
In Act 2, Scene 2, Hamlet has just finished grilling Rosencrantz and Guildenstern about their motives. They dissuade him from his anger by pointing out a band of traveling actors who has just made their way onto the grounds of Elsinore.
In a primary example of Hamlet's intellectually quick mind, the prince immediately forms an idea. There is a traditional play that many actors know called "The Murder of Gonzago." This play depicts the murder of a king.
The play's the thing
Wherein I'll catch the conscience of the King (2.2.633-34).
Hamlet pulls one of the actors aside and asks him if he can "modify" the script a little. He asks the players to memorize sixteen lines that he himself will compose and that the players are to incorporate into their act. These lines will reflect the details of Claudius and Gertrude's treachery:
Hamlet eagerly anticipates Claudius' reaction to the scene. He believes that if Claudius reacts negatively (as Hamlet is quite sure he will), that proof of the Ghost's assertions will be validated. Remember, at this point, Hamlet is still wary of trusting the Ghost. It is quite possible that the Ghost may be an evil spirit coming to him in the guise of his beloved father. However, if Hamlet gets the proof he needs, via Claudius' reaction, he will know the Ghost can be trusted.