How does Hamlet test the conscience of the king?
Actually, according to the text, Hamlet intends to "catch the conscience of the King." (II, ii, 601) This is an important distinction, because in "catching" rather than "testing" the King, Hamlet demonstrates to the audience that he is no longer wavering over whether the ghost he met in Act I was, in fact, his father, and the story and challenge of revenge that the ghost issued him was true.
In Act Three, Hamlet stages a play he calls The Mousetrap, which is the story of a King who is poisoned by his nephew as he sleeps in the garden. The villainous nephew pours poison in the King's ear -- exactly the method described by the Ghost to Hamlet concerning the method used by Claudius to kill Hamlet Senior. This performance is meant to be the means by which Hamlet catches Claudius. Here, in Act III, scene ii, just before the play is performed, Hamlet instructs Horatio in testing for King Claudius:
There is a play tonight before the King:
One scene of it comes near the circumstance
Which I have told thee of my father's death.
I prithee, when thou seest that act afoot,
. . .Observe my uncle.. . . Give him heedful note.
And when Claudius witnesses the murder in the play, he jumps up, and abruptly exits the room. Hamlet is convinced that this is evidence of Claudius' guilt, and ultimately follows Claudius with the intent to exact the revenge. He catches Claudius at prayer, however, and refuses to kill him as he is asking forgiveness of his sins. This moment is ironic however, as, in fact, Claudius is not forgiven his sins since he is unable to send his words up to heaven, but rather they "remain below."
For more on Act III, scene ii, please follow the links below.