An important consideration when approaching the question of madness in characters in drama from the Renaissance is that there were rules of a sort about using form and structure in language to signal aspects of character, and these rules applied to the playwright's use of verse and/or prose.
For example, characters speaking in a formal and regal way always spoke in verse--the poetic text written in Shakespeare's plays in iambic pentameter. Any character (especially characters in love) who uses descriptive and poetic language speaks in verse. This was simply a "rule" of drama in Shakespeare's day.
For prose, the most common usage was by the clowns, the "low-born" comic characters, but prose was also a very common choice to denote the state of mind in a character being "madness." King Lear speaks in prose during his "mad" scenes on the heath, Hamlet speaks in prose when he is onstage with characters he wishes to believe that he is "mad," and Ophelia, in Act IV, once Polonius is dead, speaks in prose.
Why all the information about prose vesus verse? Well, in Hamlet, it is the best way I know of to settle the question of Hamlet's madness. The key to the question of Hamlet's being "really mad" or not is not how he treats Ophelia or Polonius or Rosencrantz and Guildenstern. It is his use of prose when he is onstage with these characters versus his sublime and lucid use of verse when he is onstage alone with the audience and when he is onstage with Horatio (who is in on his ruse).
Hamlet is pretending to be mad, just as he said he would in Act I, scene v:
Here, as before, never, so help you mercy,
How strange or odd some'er I bear myself --
As I perchance hereafter shall think meet
To put an antic disposition on. . .
And Ophelia is quite mad indeed when we see her post-Polonius' murder in Act IV. There is, in fact, nothing at all like Hamlet's thoughtful, penetrating soliloquies delivered by Ophelia. She speaks of her "ladies" and her "coach," and sings disjointed bawdy songs. All in prose, indicating her madness. Hamlet's soliloquies are in verse, indicating the lucid, high-born beauty of his speech, and reminding the audience, when he is alone with them, that he is but "mad in craft."
For more on madness and the use of prose in Hamlet, please follow the links below.