In Hamlet, what was Hamlet's plan in staging the play within the play? Was his plan successful?
In Hamlet, Hamlet arranges for the play to be staged so he can judge the king's reaction to the murder scene within it. Hamlet is looking for verification that the ghost from act one was telling him the truth. The ghost could be what it claims to be or it could be a demon from hell sent to make mischief. The play that's staged presents a murder with a killing performed in an almost identical way that the ghost says he (Hamlet's father) was killed. If the king reacts to the production in a negative way (as indeed he does), then Hamlet will know the ghost told him the truth and he will know the ghost is really that of his father. Hamlet even arranges for an extra set of eyes to watch the king when he tells Horatio:
Even with the very comment [acutest observation] of thy soul
Observe my uncle: if his occulted [hidden] guilt
Do not itself unkennel [force from hiding] in one speech,
It is a damned ghost that we have seen,... (Act3:2)
Hamlet is seeking proof that Claudius killed his father and he gets it.
What Hamlet was trying to do by staging this play was to "prick the conscience of the king." He was trying to make the king feel bad about killing his brother (or at least to let him know that someone knew what he had done).
If this was the plan, Hamlet succeeded pretty well. As Claudius watched the play, he became agitated and eventually left. This would tend to indicate that he started to feel guilt or nervousness, or some other similarly negative emotion.
Since King Claudius left the room, it seems that Hamlet accomplished what he set out to do.