Here's the scene (Act 3, scene 3), and it's not in church; it's in the castle:
Now might I do it pat, now he is praying;
And now I'll do't. And so he goes to heaven,
And so am I revenged. That would be scann'd.
A villain kills my father; and for that,
I, his sole son, do this same villain send
O, this is hire and salary, not revenge!
He took my father grossly, full of bread,
With all his crimes broad blown, as flush as May;
And how his audit stands, who knows save heaven?
But in our circumstance and course of thought,(85)
'tis heavy with him; and am I then revenged,
To take him in the purging of his soul,
When he is fit and seasoned for his passage?
Up, sword, and know thou a more horrid hent.(90)
When he is drunk asleep; or in his rage;
Or in the incestuous pleasure of his bed;
At game, a-swearing, or about some act
That has no relish of salvation in't
Then trip him, that his heels may kick at heaven,(95)
And that his soul may be as damn'd and black
As hell, whereto it goes. My mother stays.
This physic but prolongs thy sickly days.
It's all there. Hamlet doesn't just want to kill Claudius, he wants to exact his revenge when the new king is at his worst, when he is at his grossest, his most lowly and evil. When he is drinking, or swearing, or having sex with Hamlet's mother, that's when he wants to get him. Hamlet finds Claudius at prayer and thinks that, if he kills Claudius while doing so, his soul will go to heaven rather than hell where it so justly belongs. So Hamlet puts away his sword. As Hamlet says, this only puts off the eventual and belated revenge.
Of course we later find out that Claudius could not really pray, but there is no way for Hamlet to have known this.
Yes, it may well be that Hamlet is too smart for his own good and too pricise in his actions in general and that the Ghost would have been avenged there and then, but that's a different consideration altogether.
I disagree with jseligmann. Claudius is obviously praying in this scene. He is trying to repent from his sin (killing King Hamlet). Hamlet does not kill Claudius in this scene because Hamlet is toying with Claudius's mind.
i.e Claudius knows that Hamlet is going to exact revenge over him but he does not know when and where. Hence this madness causes Claudius to ask for forgiveness from God.