What does "The Mousetrap" refer to in Hamlet?

"The Mousetrap" is Hamlet's response to his uncle, Claudius, the king, when Claudius asks Hamlet the name of the play that Hamlet has arranged to be performed at court.

The actual name of the play is The Murder of Gonzago, which Hamlet explains to Claudius:

HAMLET: This play is
the image of a murder done in Vienna. Gonzago is the Duke's
name; his wife, Baptista. You shall see anon. 'tis a knavish
piece of work (3.2.227–230)

This play is commonly referred to as the "play-within-a-play" that Hamlet asked the traveling players who recently arrived at Elsinore to perform, wherein Hamlet hopes to "catch the conscience of the King" (2.2.600).

In other words, Hamlet hopes that Claudius will reveal by his response to the play whether he has a guilty conscience, and whether he did, in fact, murder Hamlet's father as the Ghost of Hamlet's father said he did. This is why Hamlet calls the play "The Mousetrap." He hopes to catch a rat with it.

When Hamlet asked the Players to perform The Murder of Gonzago, he made a request:

HAMLET: You could, for a need, study
a speech of some dozen or sixteen lines which I would set
down and insert in't, could you not? (2.2.533-535)

There's no indication in Hamlet which of the lines in The Murder of Gonzago are the lines that Hamlet asked to be inserted in the play. Nevertheless, the play accomplishes Hamlet's objective. Claudius is upset by the play, disrupts the performance, and quickly leaves the room.

This convinces Hamlet that the Ghost was telling the truth, and that Claudius did indeed murder Hamlet's father.

HAMLET: I'll take the ghost's word for a
thousand pound! (3.2.275–276)

What does "The Mousetrap" signify about Hamlet's relationships in Hamlet?

There are a number of aspects to "The Mousetrap" and how it functions, and I am glad that your question points towards its deeper significance rather than merely focusing on how it relates to Hamlet's realtionship with Claudius. Certainly, this would be one of the central aspects concerning the play within a play. Let us remember the famous lines that Hamlet gives in his soliloquy at the end of Act II:

I'll have grounds

More relative than this: the play's the thing,

Wherein I'll catch the conscience of the King.

The Mousetrap then becomes part of the elaborate game of cat-and-mouse that Hamlet plays to try and work out whether the Ghost's words are true before he does something irrevocable like kill Claudius and ends up in hell. Yet, at the same time, in a sense, The Mousetrap is a repeated motif that is echoed throughout the entire play with Hamlet himself as the mouse. Note the number of times in which Hamlet is eavesdropped upon and observed to try and ascertain what is wrong with him. Consider the number of times he is approached by someone who was formerly a friend or lover or caring family member with a very different agenda. Rosencrantz and Guildenstern, Polonius, Laertes, Ophelia, and his own mother are all used by Claudius to try and trick and trap Hamlet in various ways. Whilst we associate The Mousetrap with something that Hamlet does to Claudius, actually, he is merely repeating what Claudius has already done many times and will continue to do to Hamlet. Hamlet is merely returning the favour.

A recent production of the Shakespeare Company of this play tried to place this element at the forefront of its delivery by setting Hamlet in some sort of police state with bugs and observation screens to highlight the way in which Claudius abused power and created some sort of Big Brother system of government. Hamlet has no peace or privacy and is constantly observed, watched and monitored. Whilst this is one extreme reading, it does help remind us that really, this play consists of many different, smaller plays with characters playing various parts and roles to try and trick Hamlet. After such an elaborate performance, no wonder Hamlet greets his death looking forward to silence: "The rest is silence." Trying to work out who is being real and who is being sincere and which person is betraying him must have been a nightmare for him. Thus we can see The Mousetrap as representing a motif that runs throughout the play.

Why is there a play, The Mousetrap, within Hamlet?

There is a play, The Mousetrap, with the play, Hamlet, because the titular character has been tasked, by the ghost of his father, with exacting revenge on his father's murderer. The ghost tells his son, Hamlet, that he was murdered by his own brother, Claudius, the man who is now the king, Hamlet's step-father, and Hamlet's uncle. Hamlet feels he must make sure his Uncle Claudius is truly his father's murderer before he can take his revenge. Therefore, when a troupe of actors comes to town, Hamlet asks them to perform this play—with a plot that is similar to the story his ghostly father told him about his own death—so he can observe his uncle's response and ascertain whether he is truly guilty. As Hamlet says, "The play's the thing / Wherein I'll catch the conscience of the king" (Act II, Scene 2, lines 566-567).