In Shakespeare's Hamlet, the first mention of madness is made by Horatio, who warns Hamlet about going off alone with the ghost of Hamlet's father.
HORATIO. What if it tempt you toward the flood, my lord,
Or to the dreadful summit of the cliff
That beetles o'er his base into the sea,
And there assume some other horrible form,
Which might deprive your sovereignty of reason
And draw you into madness? (1.4.75-80)
Hamlet rejects Horatio's advice and goes off with the ghost.
When Hamlet returns from speaking with the ghost, he ask Horatio and Marcellus to swear not to mention a word to anyone about what they've seen.
HAMLET. (to Horatio and Marcellus) And now, good friends,
As you are friends, scholars, and soldiers,
Give me one poor request.
HORATIO. What is't, my lord? we will.
HAMLET. Never make known what you have seen to-night.
HORATIO/MARCELLUS. My lord, we will not.
Hamlet then tells Horatio and Marcellus that he's going to put on an "antic disposition," and he asks them to swear not to tell anyone or imply to anyone that it's only an act.
HAMLET. But come!
Here, as before, never, so help you mercy,
How strange or odd soe'er I bear myself—
As I perchance hereafter shall think meet
To put an antic disposition on—
That you, at such times seeing me, never shall,
With arms encumber'd thus, or this head-shake,
Or by pronouncing of some doubtful phrase,
As “Well, well, we know,” or “We could, an if we would,”
Or “If we list to speak” or “There be, an if they might,”
Or such ambiguous giving out, to note
That you know aught of me... (1.5.188-198)
Putting on an "antic disposition" is generally interpreted to mean that Hamlet is going to feign madness. He's going to pretend to be insane. Does the ghost, as Horatio feared, actually cause Hamlet to go insane, or is Hamlet simply intending to act like he is insane?
To Elizabethans, "madness" includes symptoms of "melancholia"—feelings of deep sadness, depression, and hopelessness —and it's clear that Hamlet is suffering from melancholy as a result of his father's death.
In act 1, scene 2, Claudius and Gertrude tell Hamlet that his show of grief for his father's death is excessive. Claudius remarks that Hamlet's grief shows "a mind impatient" and that it's a "fault to nature, / To reason most absurd" (1.2.99, 105-106). In other words, Hamlet's behavior denotes a kind of madness, and according to Claudius, Hamlet is acting insane.
Gertrude actually accuses Hamlet of acting like he's grieving, but Hamlet assures her that his grief is sincere.
GERTRUDE. Why seems it so particular with thee?
HAMLET. Seems, madam? Nay, it is. I know not seems. (1.2.78-79)
Hamlet seems certain about the difference between real grief and feigned emotional behavior. He knows what acting is.
There's a precedent for Hamlet feigning madness. One of Shakespeare's sources for Hamlet is Amleth, Prince of Denmark from Gesta Danorum, a retelling of ancient Danish legends written by Saxo Grammaticus around 1185. In the story, Amleth pretends to be insane in order to take revenge on his uncle, Feng, who killed Amleth's father and married Amleth's mother, Gurutha.
In Shakespeare's time, "antic" also meant clownish, playful, or madcap behavior. Ophelia is the first person who appears to be a victim of Hamlet's "antic disposition." A frightened Ophelia goes to Polonius to tell him about Hamlet's strange attire and behavior.
OPHELIA. My lord, as I was sewing in my closet,
Lord Hamlet, with his doublet all unbraced,
No hat upon his head, his stockings fouled,
Ungartered, and down-gyved to his ankle;
Pale as his shirt, his knees knocking each other,
And with a look so piteous in purport
As if he had been loosed out of hell
To speak of horrors, he comes before me. (2.1.87-94)
If not for Hamlet's subsequent behavior, his appearance, as Ophelia describes it, would be "antic," or comical.
Polonius has recently been talking with Ophelia about Hamlet's professed love towards Ophelia, and that's what he has on his mind when Ophelia tells him about Hamlet's strange and frightening behavior.
POLONIUS. Mad for thy love? (2.1.95)
Polonius gets it wrong.
A little later in the play, though, Polonius might actually get it right. In act 2, scene 2, Polonius has an odd conversation with Hamlet, in which Hamlet calls Polonius a "fishmonger," asks Polonius if he has a daughter, and talks about the physical and mental deficiencies of old men.
Polonius thinks that this is somehow related to Hamlet's love for Ophelia—"Still harping on my daughter" (2.2.199). He is wrong again, but he offers clear insight into Hamlet's behavior.
POLONIUS. Though this be madness, yet there is a method in't... (2.2.216-217)
Rosencrantz and Guildenstern also seem to see through Hamlet's madness act.
ROSENCRANTZ. He does confess he feels himself distracted,
But from what cause he will by no means speak.
GUILDENSTERN. Nor do we find him forward to be sounded,
But with a crafty madness keeps aloof
When we would bring him on to some confession
Of his true state. (3.1.5-10)
Claudius, too, harbors doubts that Hamlet is truly mad.
CLAUDIUS. Love? His affections do not that way tend;
Nor what he spake, though it lack'd form a little,
Was not like madness. There's something in his soul
O'er which his melancholy sits on brood;
And I do doubt the hatch and the disclose
Will be some danger... (3.1.171-176)
Nevertheless, Claudius is a cautious man, and intends to keep an eye on Hamlet's behavior.
CLAUDIUS. Madness in great ones must not unwatch'd go. (3.1.198)
If there is any doubt about Hamlet's madness being real, however—at least to Gertrude and Claudius—those doubts are dispelled when Hamlet speaks with his mother, Gertrude, after the performance of the play-within-a-play, The Murder of Gonzago.
First, Hamlet kills Polonius, who was eavesdropping on Hamlet's conversation with Gertrude, which is, according to Gertrude, a "rash and bloody" thing to do. (3.4.30).
Then, the ghost of Hamlet's father appears to Hamlet, which firmly convinces Gertrude that Hamlet is mad, and she says so.
GERTRUDE. Alas, he's mad! (3.4.116)
However, Hamlet insists he's not mad, and he actually tells Gertrude that it's all an act.
HAMLET. ... I essentially am not in madness,
But mad in craft. (3.4.203-204)
Gertrude convinces Claudius that Hamlet is mad, and, whether Hamlet is truly mad or not, Claudius acts cautiously to limit any negative effects of Hamlet's behavior. Claudius sends Hamlet to England to be killed, and when that fails, he conspires with Laertes to kill him in a fencing match.
Some measure of Hamlet's real or feigned madness might be implied from Hamlet's conversation with one of the gravediggers, during which Hamlet helps the gravedigger to make fun of Hamlet's own madness.
HAMLET. How long hast thou been a grave-maker?
GRAVEDIGGER. Of all the days i' the year, I came to't that day
that our last king Hamlet overcame Fortinbras.
HAMLET. How long is that since?
GRAVEDIGGER. Cannot you tell that? Every fool can tell that. It
was the very day that young Hamlet was born—he that is
mad, and sent into England.
HAMLET. Ay, marry, why was he sent into England?
GRAVEDIGGER. Why, because he was mad. A shall recover his wits there; or, if a do not, 'tis no great matter there.
GRAVEDIGGER. 'Twill not he seen in him there. There the men are as mad as he.
HAMLET. How came he mad?
GRAVEDIGGER. Very strangely, they say.
HAMLET. How strangely?
GRAVEDIGGER. Faith, e'en with losing his wits. (5.1.137-154)