What is the importance of Hamlet's madness to the play? What is the importance of Hamlet's madness to the play?
I would say that your question states itself a bit over-confidently: "Shakespeare was able to make Hamlet's madness interesting". In fact, there's no real evidence in the play that Hamlet ever actually suffers from any madness at all. It's not that the play won't bear that reading - it will, and many actors have played the part like that. But you can also play it (as David Tennant did in the most recent RSC production) as if Hamlet is completely sane from start to finish.
Hamlet says to the other characters in Act 1, Scene 5:
Here, as before, never, so help you mercy,
How strange or odd soe'er I bear myself—
As I perchance hereafter shall think meet
To put an antic disposition on—
His friends are not to wonder or react if he seems mad: because he's only putting on his "antic" (mad) "disposition" (appearance). But then, by the end of the play, he apologises to Laertes, and blames the murder of Polonius on madness:
What I have done
That might your nature, honour, and exception
Roughly awake, I here proclaim was madness.
Was't Hamlet wrong'd Laertes? Never Hamlet.
If Hamlet from himself be taken away,
And when he's not himself does wrong Laertes,
Then Hamlet does it not, Hamlet denies it.
Who does it, then? His madness. If't be so,
Hamlet is of the faction that is wrong'd;
His madness is poor Hamlet's enemy.
All the events between these two quotes depend on your reading. Mad at all, or acting mad? That, I'd suggest, is the question.
It also acts as a smoke-screen for Hamlet to find the truth about his father's death. He uses it on many occasions to throw off Rosencrantz and Guildenstern, Claudius, Laertes, and even Ophelia. The act also works to some extent on his mother--especially in the scene where the two meet in her bedchambers and Hamlet sees his father's ghost again.