It is not unreasonable that Hamlet might be having second thoughts about his encounter with the apparition who claimed to be his father. Hamlet was in a highly emotionally charged state. He had never seen a ghost before. He might not necessarily believe in ghosts. His behavior is erratic when he rejoins Horatio, Marcellus and Bernardo. And the information the ghost imparted was totally astonishing. When Hamlet has regained his composure he suggests to himself that the spirit might have been the devil.
The spirit that I have seen
May be a devil; and the devil hath power
T' assume a pleasing shape; yea, and perhaps
Out of my weakness and my melancholy,
As he is very potent with such spirits,
Abuses me to damn me. (Act II, Scene 2)
If it really were the devil attempting to induce Hamlet to commit a mortal sin, there is no question that such a powerful being would be able to assume any shape he wanted and would be able to exert a potent influence on any mere mortal.
It is characteristic of Hamlet to have second thoughts, third thoughts, and any number of thoughts. His tragic flaw, as has been frequently pointed out, most notably by Samuel Taylor Coleridge, is that he thinks too much. He is frequently second-guessing himself. An excellent example is the way he stops himself from killing Claudius when he finds him alone at prayer. Hamlet says:
Now might I do it pat, now he is praying;
And now I'll do't. And so he goes to heaven,
And so am I revenged. That would be scann'd.
A villain kills my father; and for that,
I, his sole son, do this same villain send
To heaven. (Act III, Scene 3)
By "That would be scann'd," Hamlet means the question needs deep consideration. This is just the sort of thing he delights in, because he is so adept at doing it. He has developed a habit of "scanning" everything. His years at Wittenberg have given him the mental tools for doing it with precision.
Hamlet's doubts about the identity of the spirit he met on the battlement and about the information he received there lead him to staging a play titled The Murder of Gonzago which he hopes will force Claudius to betray his guilt. The trick is successful beyond Hamlet's highest expectations. Claudius is horrified when he sees the player doing exactly what he did himself, pouring poison in a sleeping man's ear. Clausius does not merely betray his guilt in his facial expressions; he creates a big scene by disrupting the play-within-a-play, calling for lights, and fleeing from the room. After this, Hamlet is convinced and decisive. It is true that he does not kill Claudius when he finds him at prayer after the play-within-a-play is disrupted; but what is important is that Hamlet has definitely decided that Claudius is going to die. He puts up his sword and says:
This physic but prolongs thy sickly days. (III.3)
Hamlet at first believes that the ghost is his father, and is quick to vow vengeance. Yet, later he worries that the ghost is in fact, the devil, trying to coax him to commit sins.
This mirrors his apparent waffling throughout the play on acting on his need to avenge his father's death. His doubts on the identity of the ghost are just like his doubts on his ability to kill Claudius.