At this point in the play, Hamlet has witnessed Claudius's reaction to the play, "The Mousetrap," and knows he is guilty of murder. When he sees Claudius at his prayers, all alone and vulnerable, not expecting an attack and not in a position to defend himself, he realizes that here is the ideal moment to avenge his father.

Yet Hamlet does not act. His reasoning for this hesitation is that his father died without having said his prayers. His father was therefore not in a state of grace—his sins were not forgiven, which is why he was doomed to walk the earth as a ghost. If Claudius dies in prayer, he will go straight to heaven. That, to Hamlet's mind, is not an equal retribution.

Therefore, once again, Hamlet hesitates.

In a bit of dramatic irony, we as the audience know that Claudius is incapable repenting of the murder: he is glad he did it, because he likes the power he has gained as a result. Therefore, his soul isn't really clean in the eyes of God—but Hamlet doesn't know this.

In Act 3.3 of Shakespeare's Hamlet, Hamlet tells the reader why he doesn't kill Claudius and achieve his revenge:

Now might I do it pat, now 'a is a-praying,

And now I'll do't--an so 'a goes to heaven,

And so am I revenged.  That would be scanned.

A villain kills my father, and for that,

I, his sole son, do this same villain send

To heaven.  (Act 3.3.73-78)

Hamlet assumes Claudius is confessing his sins.  According to Hamlet's Catholic beliefs, Claudius' soul would be wiped clean and he would be forgiven and if killed, would go straight to heaven.  Claudius is not confessing his sins, but Hamlet assumes he is.  Thus, he chooses not to kill him and send him to heaven.

This, of course, is probably the climax of the play.  When Hamlet makes this decision he is playing God--dealing with human salvation, not revenge.  Salvation is God's business, not Hamlet's.  Because Hamlet walks away, innocent people die, including Hamlet himself.