In Act 3 , scene 1, Hamlet confronts his girlfriend, Ophelia. He is angry at her and women in general, and his words are sometimes wild and sometimes brilliant and it all seems quite crazy and frightening to her. But this is all part of Hamlet's grand plan. He wants to be a disturbance, to cause trouble... trouble that is meant to worry, upset and ultimately set up King Claudius.
Hamlet knows that if he scares Ophelia, she will tell her father, Polonius, and that Polonius (an adviser to Claudius) will relate this to the King. It's an elaborate trap Hamlet is setting for Claudius. Hamlet is moving the pieces as if he were playing a deadly game of chess which will culminate in a play that will soon be seen by Claudius...a play that will enact the murder of Hamlet's father. This is the play that Hamlet believes will "catch the conscience of the King"... (the last line of Act 2).
Hamlet doesn't just want any old revenge; he wants Claudius to suffer and to feel his guilt first before he is killed.
I believe what you are referring to is the so-called "silent interview" which takes place in Act Two, Scene One of Hamlet. In this scene, Ophelia appears and reports to her father, Polonius, that she has just had an inexplicably strange encounter with Hamlet. Hamlet appeared to Ophelia partially unclothed, looking "as if he had been looséd out of hell / To speak of horrors." Polonius interprets this behavior as a sign that Hamlet has gone made out of unrequited love for Ophelia and intends to deliver this gossip to the king.
Arguably, this encounter holds dramatic value because it shows the devious ways in which Hamlet is manipulating those around him. Hamlet knows that Ophelia will tell all of her concerns to her father, and that Polonius, in return, will whisper this information into the ear of the king. By mimicking the appearance of a distraught lover, Hamlet can pass off his otherwise suspicious behavior and plot for revenge as mere heartache. It lightens the investigative interest that the king may otherwise take in Hamlet's activities and makes him seem like less of a threat. This is particularly dramatic since we know that the opposite is true; in planning to kill the king, Hamlet is the ultimate threat.