Hamlet decided to feign madness in order to gain an advantage against Claudius and give cover to his machinations. Hamlet's performance was useful to help mask his purpose of finding out the facts as well as giving observers a reason to explain his continued depression and emotional distance.
The excuse that he is mad with grief or mad with rejected love for Ophelia (Polonius forbade Ophelia to reciprocate Hamlet's expressed love since he is the Prince) also gives Hamlet the opportunity to make statements to his mother and Claudius that he otherwise could never have made. He does this, as he says, to uncover signs of their guilt.
There is a school of thought that actually argues that in spite of his protestations to "feign" madness, Hamlet actually did become slightly unhinged due to the terrible situation he found himself in. However, it is clear that in his mind, at least, he chooses to feign madness to consider his position and gather information to try to prove if the Ghost's words were true or not. One of the many doubts that plagues Hamlet is if the Ghost is actually the ghost of his dead father or a denizen of hell sent to taunt him and trick him into committing regicide - a very serious crime in those times. His madness allows him to assess where both he and others stands, and to act accordingly.
Immediately after his first meeting with the Ghost, Hamlet swears Marcellus and Horatio to secrecy and indicates that he is thinking of pretending to be mad. Throughout the play his mental condition is a matter of increasing concern, mystery, speculation, discussion, and debate.
Hamlet knows that the King is a murderer and a usurper. He intends to assassinate the King. He knows his father’s ghost is haunting the castle. His mother may have been involved in her husband’s murder. Any of these secrets could cost him his life. His facial expressions, body language, and possible slips of the tongue might give him away. Claudius will continue to pry into his mind and his very soul. His encounter with the Ghost has changed him into a different person, and this is sure to be noticed by the King.
Hamlet’s foresight proves correct. Claudius uses his cunning mind to try to understand his stepson. Like many villains, today as well as in yesteryears, he is parasitical: he specializes in analyzing people in order to manipulate them. He uses Polonius, Ophelia, Rosencrantz, Guildenstern, and even Gertrude to spy on Hamlet. Unlike them, he is not convinced that his stepson is really mad. He says:
There's something in his soul,
O'er which his melancholy sits on brood,
And I do doubt the hatch and the disclose
Will be some danger . . .
That wonderful and characteristically Shakespearean metaphor shows the extent to which the King is trying to pry into the recesses of Hamlet’s soul and suggests the difficulty that Hamlet must be experiencing in dissimulating. He plans to kill Claudius--and that is precisely what Claudius suspects. Hamlet cannot hide. Being a prince and heir-apparent keeps him in the spotlight. Claudius says he wants to keep him
Here in the cheer and comfort of our eye,
Our chiefest courtier, cousin, and our son.
And Ophelia describes him as
The courtier’s, soldier’s, scholar’s eye – tongue – sword,
Th’ expectancy and rose of the fair state,
The glass of fashion and the mold of form,
Th’ observed of all observers, quite, quite down!
We all invent “personas” to represent ourselves to the world at large, and in emergency situations we may invent new personas. Hamlet is really doing the same thing that the fugitive Edgar does in King Lear: Edgar knows he cannot hide but must reinvent himself. Rosalind does something similar in AsYou Like It.
To be the subject of such ceaseless scrutiny could drive anyone mad — especially an introvert like Hamlet who naturally values his privacy. He cannot even feel safe with Ophelia or with his mother. He cannot really be sure that the four men who actually saw the Ghost will keep it a sworn secret. By acting insane he can present a false persona which Claudius may find impossible to penetrate.
As we go through life we encounter many people who try to figure out what makes us tick. Some may be only curious, while others may, like Claudius, be very dangerous. We can learn a practical lesson about humanity from Shakespeare’s Hamlet. Watch out for inquisitive people! They may have ulterior motives. We empathize with Hamlet, because we have often been made to feel uneasy or suspicious, if not resentful, by self-appointed mind-readers. It is ironic that Hamlet’s inquisitors are inferior to him in intelligence. He makes Polonius, Rosencrantz and Guildenstern look ridiculous with their efforts to understand the workings of his mind. He even makes Claudius—by far the cleverest inquisitor of all—look ridiculous. Claudius is trying to draw Hamlet out; instead Hamlet draws Claudius out.
Hamlet decides to "act" mad because it's going to get him what he wants--or so he thinks. We know he's an accomplished actor (his conversation with the Players reveals this), and we know everyone gives him plenty of license because they're not sure of his mental state. He continues his plan in the face of obstacles, but it does not, of course, really get him what he wants in the end. Ophelia is dead, Polonius is dead, Laertes is dead, his mother is dead, and he is dead. The fact that Claudius is also dead somehow doesn't satisfy in the way it should have. If there is a shift from feigned to true madness, I suspect it's when Hamlet realizes he is being sent to England to his execution or when he sees Ophelia's body being brought to her grave.
His madness (as he calls it when he speaks to Horatio and directs him not to pay attention to anything "crazy" he does) is supposed to help Hamlet study Claudius' actions, speech, habits, all to discern whether or not the Ghost has told Hamlet the truth or duped him into believing a falsehood, since creatures from the "other world" aren't always trustworthy. When characters believe others are crazy, they tend to say and do more than they would otherwise, since crazy people don't pay attention, and even if they do, who would believe them if they told what they heard?
The astute question of how madness is defined is, indeed, cogent. Perhaps Hamlet uses his mad outbursts as a way of venting his disgust for his mother, his antipathy for Claudius, and his frustration and repulsion regarding the "rotten" court of Denmark.
His feigning of madness can also serve to act as a smokescreen so that he can learn more about the chicanery of King Claudius as he does, for instance, with Guildenstern and Rosencrantz who end up revealing more than they intend.
It is interesting to study Hamlet as a dynamic character and to try to determine if he does cross the line from feigned madness into the real thing. And how is his "madness" defined? Often it seems to be only an exaggerated state of what he is really feeling/thinking. He never becomes another person; he only becomes "more Hamlet." By being "mad," he is freed of the need to control his emotions and his behavior.
Whether Hamlet crosses the line between sanity and madness, again, seems to depend on the definition of madness. At what point does being depressed become more than being depressed? When addressing a skull? When jumping into Ophelia's grave? When a young prince who had once been gentle and philosophical comes to embrace the cold, clever, premeditated murders of Rosencrantz and Guildenstern, has he crossed a line somewhere in his own mind?
One aspect of Hamlet that is often overlooked is the fact that it is a revenge play in the tradition of Shakespeare's day. Revenge plays were enormously popular with Shakespeare's audience, and Hamlet incorporates all the elements of the revenge play genre: murder, the eventual identification of the murderer, various obstacles that impede the avenger from gaining satisfaction--and madness. Madness was a basic staple in revenge plays. Hamlet's madness fit the bill.
Shakespeare being Shakespeare employs Hamlet's feigned madness in creating one of the great ironies of the play. In the end, it is Ophelia who becomes truly mad.
There are several reasons that Hamlet decides to feign madness. The first of which is to buy some time while gathering evidence that Claudius is indeed the murderer of the his father. It seems that Hamlet's emotions are quite apparent to all. Gertrude and Claudius both previously remarked on his melancholy following his father's death. It is necessary for Hamlet to mask his new-found suspicion of Claudius. What better way to do it than to pretend to be mad.
Also his pretense enables Hamlet to determine who is truly his ally and who is not. He first tries out his mad act on Ophelia, who immediately tells her father, who immediately reports to Claudius. Hamlet is now able to determine that Ophelia is not a confidante.
Revenge against his mother's husband and his uncle is an act that Hamlet does not relish. He needs justification, evidence, and a great deal of prompting to kill the King.
There are several theories about what Hamlet hopes to gain by pretending to be mad. His feigned madness is part of his plan to determine his uncle's guilt in murder. He believes that if people believe him to be mad due to his grief over his father's death, he can prod them more and provoke them into revealing the truth; in a sense, he plays the role of the Fool in King Lear. He says many things to his mother and Claudius, trying to ferret out information, and they are always making comments such as, "Ah, he is mad!"
Keep in mind that although his father has indeed died and his uncle has married his mother and is now the king, the only "proof" that Hamlet has that Claudius murdered his father is the word of the Ghost of Hamlet's father. Killing a king is regicide and not to be entered into lightly. So Hamlet's hesitation in carrying out the vengeance sought by the Ghost is understandable particularly in light of his Protestant religious beliefs. He must be sure.
The play leaves open the question of whether Hamlet truly is mad or whether he just puts on an antic disposition. Dover Wilson in his book "What Happens In Hamlet" (it is a bit expensive for a paperback written in the mid 30s but it is available in Google books) claims no less than 6 times Hamlet swerves into actual madness. He also claims that the swearing scene at the end of 1.5 where Hamlet claims to put on an antic disposition was a ruse to cover his true madness. I think Wilson is excessive in pushing what is deliberately ambiguous. In the text, Shakespeare is careful to balance the text both for and against feigning madness.
Having said that, the purposes are varied. In the original Amleth legend the title character acts simple minded. This serves the purpose of eliminating Amleth as a threat to the uncle king. Here, I think the intent is the same, but the irony is that Hamlet's antics draw suspicion rather than allay them.
From a thematic perspective the madness issue provides a dichotomy between image and reality. And from a theatrical perspective it gives the actor playing Hamlet a much wider range of acting. Madness provides an additional dimension to the character that can be played as the actor and director see fit.
Hamlet pretending to be mad is part of his plan to get revenge for the murder of his father. Whether or not it was his best option for his revenge is really hard to say due to the circumstances in which Shakespeare wrote it. Looking back upon it now, with a contemporary viewpoint, since it did work, it was successful.