Hamlet is so brutal to Ophelia because he cannot trust her. In Act 3, Hamlet and Ophelia are on stage together for the first time. Sadly, however, their meeting is a set up, devised by Claudius and Polonius to determine the cause of Hamlet's seeming insanity. Polonius is trying to prove that Hamlet is insane because he is heartbroken after Ophelia, as Polonius instructed, broke up with him. Claudius suspects that Hamlet may know something about the death of his father, and is worried that Hamlet may suspect Claudius of foul play.
When viewed in this light, it becomes clear that much of Hamlet's anger is directed not at Ophelia but at the eavesdroppers. While he tells Ophelia repeatedly to "get thee to a nunnery," he calls Polonius a fool and threatens Claudius' life. He seems to be well aware that the conversation he is having with Ophelia is not a private one.
Ophelia begins the meeting by returning the tokens of love that Hamlet had given her. Hamlet reacts with pain and astonishment. He immediately asks Ophelia where her father is, and she lies, telling him that Polonius is at home. Hamlet knows then that Ophelia has chosen to side with her father. Although Ophelia is not trying to hurt Hamlet, she does so by being so obedient to her father.
We can easily see why Hamlet is so cruel. Here he is with a murdered father, his mother married to the man who murdered him, and his girlfriend breaking up with him and reporting to her father all that he says and does. Hamlet reacts as many young men would--with cruel words designed to make Ophelia suffer as he is suffering.