He was not really insane, but he was deranged. He beleived that he should be king, which was irrational. He might have hallucinated the dagger, or it might have been the witches playing with him. He might really have seen Banquo's ghost. It wouldn't be the first ghost in Shakespeare either!
I'm not a psychologist or a psychiatrist, but I don't believe Macbeth was insane. Macbeth knew what he was doing at all times. He debated over whether or not to kill Duncan until his wife talked him into it. When he committed the murder, he was quite shaken and extremely nervous, as any person might be expected to be. He fears that Banquo suspects him of killing Duncan, and according to what Banquo says at the opening of Act 3, he had good reason to think Banquo is on to him. He hires murderers to kill Banquo. He goes to the witches a second time because they were right the first time they spoke with him. Odds are they might be right again. He fears that Macduff is his enemy and since he can't get ahold of Macduff, and one way to really hurt a person is to hurt those he loves, Macbeth has Macduff's family killed. These are all terrible deeds, but they are rational deeds. He is paranoid and he is ambitious, but he is not insane. In the last act, when he encounters Macduff finally in sc. 8, Macbeth tells Macduff that he cannot be killed by any man born of woman and he doesn't want to fight Macduff and kill him since he's killed enough of the Macduff family already. This is a very sane sounding speech for Macbeth to say because it shows some logic and some remorse. Macbeth's wife succumbed to her guilt, went insane: walks, talks in her sleep, and continually washes imaginary blood from her hands.
I do not think we can always determine what is insane or not in fictional literature. The best one to answer this question would be the author.
However, I do not believe murder or listening to witches is rational. Many who murder in cold blood can sound remorseful and repentant. That does not make them rationale or repentant.