Madness in Hamlet can be seen in two different lights--real madness, and feigned madness. Hamlet, in order to more safely investigate the alleged murder of his father, feigns madness, and puts on an "antic disposition" in order to throw people off. He wants people to think he's acting crazy all of the time, so that when he does come close to getting information and clues, no one will be alarmed at his snooping and questions. Instead, they'll just pass it off as "that's just crazy Hamlet, doing kooky things again." His madness does serve that purpose for a while, but the king still gets spooked and sends out death order to his friends on their journey. So, it doesn't work too well. In the meantime, he manages to alienate his girlfriend, mother, friends and all people in the castle. He asks bizarre questions, makes jests and jabs at people that show his displeasure, and is very alarming.
Compare this to Ophelia's true madness. Used to being told what to do by her father and brother, and totally reliant on their guidance and management of her life, she goes truly insane when they disappear. Laertes, on a journey of self-discovery, and her father Polonius killed by her ex-boyfriend, she is left alone and directionless in the world. First of all, Hamlet cruelly breaks up with her (after her father tells her to end it also), then he kills her father. He is merely smacked on the hand and shipped away instead of being punished for it too. Laertes is gone, and she has no family left. All of these events combine to break her mind; she walks aimlessly about, chanting nursery rhymes and sing-songs. She can't get a grip, and in that state, dies.
Hamlet's feigned madness is merely games, and seems rather insulty and trivial when compared with Ophelia's tragic descent into lunacy. Shakespeare has both characters display these "mad" traits, but the results of the true madness in Ophelia are very sad, brought on by real and repeated tragedy in her life, and lead to her death. Hamlet's games are simply infuriating to everyone around him, and allow him to dodge repercussions for truly harmful actions. I hope that those thoughts help to get you started; good luck!