I think it is safe to say that Hamlet is aware of what he is doing most of the time, though in some situations it could be argued that he acts irrationally.
At the start of the play, when Hamlet has met the ghost of his father, he plots to prove Claudius is the old king's murderer. (In fact, Old Hamlet has charged Hamlet with the task of avenging his death.) With this is mind, Hamlet says he will pretend madness, expecting that people will let their guard down if they believe Hamlet is insane. He insists that he acts crazy only to discover the truth of his father's murder, but really knows what is going on.
I am but mad north-north-west: when the wind is southerly I know a hawk from a handsaw. (II, ii, 272)
Hamlet acts crazy with everyone, even poor Ophelia. He is extremely unsure who he can trust, so he puts on "an antic disposition" in order to throw everyone off. He cannot trust Ophelia as she is a dutiful daughter, and Polonius is beyond loyal to the new king, Claudius. Ophelia has no part in the scheming. She is an innocent caught up in the deceit and evil that permeates the castle. Hamlet's pretended insanity, and then her father's death, ultimately drive Ophelia insane.
It could be argued, however, that when Hamlet finds someone behind the arras (curtain) in his mother's room, thinking it is his step-father, he lunges forward and stabs the man there, though it is, in truth, Polonius, Ophelia's father.
His "impetuous" act, as his mother calls it, could be seen as a blind, passionate action that would probably be classified today as manslaughter. It is not premeditated in that moment (though Hamlet is trying to avenge his father's death at some point).
Hamlet seizes this moment, believing that Claudius has been having "incestuous" sex with Gertrude, and will die with sins upon his soul if he dies right then. Hamlet acts rashly, and commits murder.
Bottom line: I believe Hamlet knows what he is doing, though it would be easily argued that he loses control temporarily, mistakenly killing Polonius.