There is littl mystery, in Hamlet's mind, as to who killed his father. He becomes convinced, partly because of the hasty marriage between Gertrude and Claudius, that Claudius is the culprit. It would be correct to say that Claudius behaves mysteriously inasmuch as he is hiding the fact that he killed the king and is secretly trying to get rid of Hamlet because he feels Hamlet is a threat.
From the perspective of all the other characters, it is Hamlet who behaves mysteriously. In their eyes, he goes from brooding and insular to manic and unpredictable. His actions alienate Ophelia, frighten Claudius, anger Laertes, and worry his mother. The mystery, for the reader/audience is not necessarily why Hamlet is acting this way because we know that Hamlet is "acting" in these ways because he doesn't want to compromise his plan for revenge. In fact, the mystery (for the reader/audience) is just how and when (finally) Hamlet will get his revenge. Hamlet is often called a revenge play but it might just as well be called the "delay of revenge" play because that's what the play is essentially about.
Now, the mystery is: Why does Hamlet delay?
From the reader's perspective, if Hamlet kills Claudius right away, the play is over. That may seem ridiculous but consider that the play is about Hamlet's philosophizing about revenge more than it is about the act of revenge. And for someone as thoughtful and overly analytical as Hamlet, this will take some time. Hamlet delays his revenge because he wants it to be righteous. In Act 3, Hamlet is about to kill Claudius but he hesitates because Claudius is praying. To kill a praying man would be too good for Claudius, Hamlet thinks:
But in our circumstance and course of thought;
'tis heavy with him, and I am then revenged,
To take him in the purging of his soul,
When he is fit and seasoned for his passage?
Hamlet also delays because he wants to expose the sins of Claudius to as many people as possible. In thinking over his plan for revenge, Hamlet determines that his revenge must be righteous but also dramatic. He wants to embarrass Claudius in addition to avenging his (Hamlet's) father. Hamlet devises a way to do this with the "play within a play." Hamlet over-thinks things so much that he feels he needs the play to prove Claudius' guilt because his father's ghost could also be the Devil.
I know my course. The spirit that I have seen
May be a devil; and the devil hath power
t'assume a pleasing shape; yea, and perhaps
Out of my weakness and my melancholy,
As he is very potent with such spirits,
Abuses me to damn me. I'll have grounds
More relative than this. The play's the thing
Wherein I'll catch the conscience of the king. (II.ii.593-60)
Even here, Hamlet is playing the role of philosopher and lawyer. He wants the exposition (complete description) of Claudius' guilt to be unquestionable. Thus, he will have the "grounds" for doing so when he (and others) see Claudius' reaction to the play.
Hamlet acts mysteriously to keep the other characters off guard. Hamlet delays because he wants his revenge to be thoughtful and monumental. The mystery, revealed throughout the course of the play, is why Hamlet delays and then how his extravagant planning will play out.