I believe Hamlet was extremely anxious over his father's death. He did feel compelled to avenge his father's death. He is driven by his rage and anguish over losing his father. That alone is enough to make one mad. Nevertheless, I believe Hamlet held it together well. Under the circumstances, any normal person would feel anxiety and mental and emotional stress. His father has been murdered. That is enough to push one over the edge, but to add to that, his father has been murdered by his uncle, and his uncle has married his mother. Under these horrific circumstances, anyone would suffer mental anxiety.
I believe Hamlet held it together well. He cautiously reenacted his father's murderous scene, searching for his uncle's guilt. He wanted to be sure that Claudius had killed his father. He is carefully contemplating what his further actions should be. This type behavior is done under well-controlled thinking processes.
If there was a time Hamlet appeared to be mad, it could have been when he stabbed the man behind the curtain, not knowing for certain whom he was killing. But then again, he was thought it was his Uncle Claudius, and his death would have avenged his father's death. This would not be calculating. If he had killed the right man, this would have been justice.