Act 3, scene 1 is the first time in Shakespeare's Hamlet that the audience sees Hamlet and Ophelia together. There's very little frame of reference for what happens in this scene, except for Polonius and Laertes advising Ophelia to stay away from Hamlet in act 1, scene 3 and Ophelia's story to Polonius, in act 2, scene 1, when Hamlet puts on his "antic disposition" and frightens Ophelia while she's sewing quietly in her room.

When Hamlet enters in act 3, scene 1, he seems thoughtful, rational, and notably calm throughout his "To be, or not to be" soliloquy, particularly compared to his evident state of mind in his three previous soliloquies.

In his "O, that this too, too sullied flesh would melt" soliloquy in act 1, scene 2, Hamlet is grieved by his father's death and appalled by his mother's marriage to a "satyr," his uncle Claudius.

In the second soliloquy, "O all you host of heaven!" after he's seen the ghost of his father, Hamlet is understandably agitated and animated. He vows to avenge his father's murder and denounces Claudius as a "smiling, damned villain" (1.5.111).

In the "O, what a rogue and peasant slave am I!" soliloquy, Hamlet severely berates himself—and at considerable length—for his cowardly indecision and for his utter failure to do anything to avenge his father's murder.

Up to this point in the play, Hamlet has "played words" and sparred intellectually with Polonius, Rosencrantz, and Guildenstern, but he's been relatively cordial with all of them, and hasn't lashed out verbally at any of them. The only person to whom Hamlet has spoken even the least bit sharply is himself.

What happened in the few minutes between the overwrought "O, what a rogue and peasant slave am I!" soliloquy and the quietly controlled "To be, or not to be?" soliloquy? What happens to Hamlet when Ophelia appears in the scene that causes him to change his demeanor so dramatically and react to her so emotionally?

One explanation for Hamlet's behavior towards Ophelia might be that he's aware that he and Ophelia are being watched, probably by Claudius and Polonius, so he puts on his "antic disposition" for them. Does Hamlet simply go too far with his "antic disposition" and say more to Ophelia than he truly meant to say?

As a corollary to that, Hamlet might be upset with Ophelia for betraying him by participating in Claudius and Polonius's scheme to listen in on their conversation. This might have prompted his "Are you honest?" question to her.

Does Hamlet think, perhaps, that Ophelia is trying to end their relationship by returning the "remembrances" to him? This might explain his emotional outbursts and verbal abuse of Ophelia, and his "I did love you once" and "I loved you not"—although "get thee to a nunnery" seems a bit of an overreaction.

If Hamlet knows that Claudius and Polonius are eavesdropping on his conversation with Ophelia, he might simply be taking advantage of the opportunity to demean Polonius as a fool and to threaten Claudius with "Those that are married already—all but one—shall live" (3.1.156–157).

The next time that Hamlet and Ophelia see each other is in the play-within-a-play scene, act 3, scene 2, which happens not more than a few hours after the "get thee to a nunnery" scene. By then, everything seems to be forgiven between them. They banter with one another and engage in some suggestive wordplay, but not a word is said about Hamlet's overly harsh treatment of Ophelia earlier that day.

Could Ophelia have known exactly what Hamlet was doing in the "get thee to a nunnery" scene and simply played along with his "antic disposition" act? Could Ophelia have made up the story about Hamlet coming into her room and frightening her with his disheveled clothes and strange behavior to help Hamlet mislead Polonius and Claudius about the reason for his "madness"?

One question leads to another in Hamlet, and there's no definitive answers to most of those questions.

There's much debate as to why Hamlet is so brutal towards Ophelia. First, she enters as he is contemplating suicide, so he is not in a good state of mind. Second, we know this is not the first time he has been brutal, for as she gives back the gifts, she says 

Rich gifts wax poor when givers prove unkind.

 I say, we will have no more marriages. Those that are married already, all but one, shall live.

Hamlet thus has many reasons for speaking brutally to Ophelia: his own suicidal, frenzied state of mind, his distrust of anyone at court, his anger at and repressed sexual desire for his mother, a conviction that all of humankind is corrupt (which inspires him not to want any more marriages), and possibly a desire to protect Ophelia from the coming upheaval as he avenges his father's death. 

Hamlet is so brutal to Ophelia because he cannot trust her.  In Act 3, Hamlet and Ophelia are on stage together for the first time.  Sadly, however, their meeting is a set up, devised by Claudius and Polonius to determine the cause of Hamlet's seeming insanity.  Polonius is trying to prove that Hamlet is insane because he is heartbroken after Ophelia, as Polonius instructed, broke up with him.  Claudius suspects that Hamlet may know something about the death of his father, and is worried that Hamlet may suspect Claudius of foul play.

When viewed in this light, it becomes clear that much of Hamlet's anger is directed not at Ophelia but at the eavesdroppers.  While he tells Ophelia repeatedly to "get thee to a nunnery," he calls Polonius a fool and threatens Claudius' life.  He seems to be well aware that the conversation he is having with Ophelia is not a private one.

Ophelia begins the meeting by returning the tokens of love that Hamlet had given her.  Hamlet reacts with pain and astonishment.  He immediately asks Ophelia where her father is, and she lies, telling him that Polonius is at home.  Hamlet knows then that Ophelia has chosen to side with her father.  Although Ophelia is not trying to hurt Hamlet, she does so by being so obedient to her father.

We can easily see why Hamlet is so cruel.  Here he is with a murdered father, his mother married to the man who murdered him, and his girlfriend breaking up with him and reporting to her father all that he says and does.  Hamlet reacts as many young men would--with cruel words designed to make Ophelia suffer as he is suffering.