Hamlet is dealing with a complex dilemma. He think his uncle, who has recently married his mother, also murdered his father. The main evidence he has for this is that his father's ghost has appeared to him. The ghost of Hamlet's father describes how Claudius came upon him while he was sleeping and poured poison in his ear to kill him. Because Hamlet was already angry that Claudius married his mother, Hamlet was willing to believe the ghost's account, but the word of a ghost is not really credible proof of murder. In this period, ghosts tended to be regarded as evil, and certainly not as trustworthy and credible witnesses. In fact, they were more likely to be instruments of the Devil leading people astray. Thus Hamlet seeks a way to ascertain Claudius' guilt.
When the players arrive, he decides to have them perform a play which mimics Claudius' murder of his father, thinking that Claudius in his reaction to the play will display his guilty conscience. Hamlet thus explains that the play will give him better evidence of Claudius' guilt than the ghost, saying:
... I'll have grounds
More relative than this. The play’s the thing
Wherein I’ll catch the conscience of the king.