In Shakespeare's Hamlet, Hamlet believes Claudius is doing more than just praying when he comes upon him in Act 3.3:  he thinks he is confessing. 

The play features Catholic theology more than it features Protestant.  The Ghost wandering around until his sins are burnt and purged away is like the Catholic belief in purgatory, not like anything Protestants believe in:

I am thy father's spirit,

Doomed for a certain term to walk the night,

And for the day confined to fast in fires,

Till the foul crimes done in my days of nature

Are burnt and purged away....  (Act 1.5:9-13)

This is purgatory, or something close to it. 

Applying Catholic theology to Hamlet's refusal to kill Claudius at prayer, then, suggests that Hamlet is reacting to the belief that if Claudius is confessing, his soul would already be purged and he would be guiltless before God and would therefore go straight to heaven. 

It is no accident that Claudius in his prayer stresses forgiveness. 

...Whereto serves mercy

But to confront the visage of offence [what is mercy good for if not to meet sin face-to-face]?

And what's in prayer but this twofold force,

To be forestalled ere we come to fall,

Or pardoned being down [what is prayer for if not to stop us from sinning, or to forgive us when we do]?    Then I'll look up.

My fault is past.  But, O, what form of prayer

Can serve my turn?  'Forgive me my foul murder'? 

That cannot be, since I am still possessed

Of those effects for which I did the murder--

My crown, mine own ambiton, and my queen.

May one be pardoned and retain th' offence?  (Act 3.3.36-56)

Claudius stresses confession and mercy and forgiveness, and that's what Hamlet suspects.

The problem with Hamlet's decision to not kill Claudius because he doesn't want to send him to heaven, is that that places Hamlet in the role of God.  In other words, Hamlet is playing God, here.  He is messing where he doesn't belong.  And the consequence for his actions is the blood bath at the end of the play.  This is probably the climax of the play, the point at which Hamlet and so many others are doomed. 

Ironically, Claudius does not confess.  He cannot.  He is at least intellectually honest enough to realize that he cannot be forgiven when he hasn't repented, when he hasn't given up that which he got by sinning in the first place.  Hamlet could have revenged his father by killing Claudius, and not sent him to heaven. 

Hamlet comes across Claudius while he is praying.  He believes that if he kills Claudius while he is praying, it will increase his odds of getting into heaven.  Praying is a good deed; if you die while praying, it might help your case in heaven.  Hamlet hates Claudius; he does not want him to go to heaven, but to hell.  So, he decides that he will wait until Claudius is doing something horrible and awful, and that will ensure that he will get sent straight down to hell.  The quotes for this scene can be found in Act Three, Scene Three.  Here's just a portion of them:

"Now might I do it pat, now he is praying;
And now I'll do't. And so he goes to heaven,
And so am I revenged. That would be scann'd.
A villain kills my father; and for that,
I, his sole son, do this same villain send
To heaven.(80) ...
When he is drunk asleep; or in his rage;
Or in the incestuous pleasure of his bed;
At game, a-swearing, or about some act...
Then trip him, that his heels may kick at heaven,(95)
And that his soul may be as damn'd and black
As hell, whereto it goes."

The reason that he gives for not killing Hamlet is founded in religious beliefs about heaven and hell, and about how our behavior here on earth influences where we go after we die.  In reality though, I think that Hamlet was just procrastinating the revenge.  He wasn't ready, at that point in the play, to actually do the deed.  So, he had a handy excuse ready, and used it gladly.  I hope that helped; good luck!