In Shakespeare's play Hamlet, the hero may have be depressed, melancholy, frustrated, and confused, but there is not much evidence that he might be really mad. It would be highly unusual for an author to invent a character who was both pretending to be mad and mad in fact. Hamlet himself should be his own best witness. In Act I, Scene 5 he makes Horatio and Marcellus swear:
Here, as before: never, so help you mercy,
How strange or odd some'er I bear myself --
As I perchance hereafter shall think meet
To put an antic disposition on,
That you, at such times seeing me, never shall,
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[hint] That you know aught of me--this do swear,
So grace and mercy at your most need help you.
This not only shows that Hamlet intends to pretend to be mad but that he has a very clear and intelligent mind. He can foresee that his astonishing new supernatural knowledge is bound to affect his behavior towards the King, who is already extremely suspicious of his intentions and keeping him a virtual prisoner at Elsinore, where he spies on him and encourages others to do so as well. By acting mad, Hamlet hopes to disguise his real thoughts, feelings, and intentions from a cunning adversary and friendly enemies such as Polonius and his two schoolfellows Rosencrantz and Guildenstern.
Further evidence from Hamlet himself that he is not mad comes in Act 3, Scene 4 when the Ghost appears once again to his son but cannot be seen or heard by Gertrude, who believes this is proof her son is mad.
This is the very coinage of your brain,
This bodiless creation ecstasy
Is very cunning in.
My pulse as yours doth temperately keep time,
And makes as healthful music. It is not madness
That I have uttered. Bring me to the test,
And I the matter will re-word, which madness
Would gambol from.
And a bit later in that same pivotal scene, he warns his mother not to let her husband King Claudius
Make you to ravel all this matter out,
That I essentially am not in madness,
But mad in craft.
Many of the characters in the play are led to believe that Hamlet is insane, but this proves nothing. Hamlet wants them to think he is insane. The "antic disposition" he assumes to play mind games with the King and his courtiers is sometimes so convincing that the audience is also deceived.
Two problems in your question first. "Was" isn't proper because Hamlet "is". Every time Hamlet is performed, watched, or read it "is". Second, Hamlet isn't a documentary. It is fiction. There was no real Hamlet that awaits psychiatric discovery. Whether Hamlet is mad depends on the how it is interpreted. Shakespeare in the text is ambiguous about Hamlet's mental state. In the source material "Amleth", the main character plays a simpleton to have his uncle think that Amleth is not present a threat to him. In Hamlet, Shakespeare presents the idea of Hamlet's madness. One reason is thematic: image and reality. Is Hamlet mad or is he acting mad? Within the context of the play is there a difference? Another reason is acting flexibility. The specter of madness gives the actor playing Hamlet a much wider acting range. But again does the actor play the part to be mad or just act mad? There is no wrong answer. Conversely, in J. Dover Wilson's book, What Happens In Hamlet, Wilson claims that Hamlet actually goes mad on six different occasions during the play. That is of course his interpretation.