In addition to the many mixed emotions Hamlet is dealing with, there are a few other interpretations of this scene that are worth noting.
The Kenneth Branaugh film depiction of the play suggests that Hamlet knows that he is being spied on at that very minute by Claudius and Polonius. I think this interpretation has a lot of validity - and makes his behavior even more accounted for.
When Hamlet asks Ophelia in line 103, "Ha, ha! Are you honest?" the footnote suggests he means to not only ask if she is at-that-moment telling him the truth, but is she also "chaste" and "modest." The interpretation here is that Hamlet and Ophelia have already carried on a sexual relationship - but that Ophelia is hiding this truth from her father and everyone else. If you look at it this way, it is almost like Hamlet is saying, "Are you lying to me about your true feelings just like you've been lying to your father about your virginity?" Perhaps Ophelia hasn't directly lied. But she's not living in complete honesty with anyone (including herself) so Hamlet has reason not to trust her.
Hamlet is angry with his mother, Claudius, and Polonius more than he is with Ophelia. He does get angry with Ophelia after she lies to him. He asks, "Where is your father?" and she answers, "At home, my lord," an obvious lie.
Hamlet knows that Ophelia is spying on him for her father. He knows that whatever he says will be heard through the grapevine of Denmark: Ophelia will tell her father; her father will tell Claudius and Gertrude. So, Hamlet is acting angry and crazy not only to misdirect the King's spies, but also to vent his anger toward his mother's incestuous relationship with her husband's murderer.
Get thee to a nunnery: why wouldst thou be a
breeder of sinners? I am myself indifferent honest;
but yet I could accuse me of such things that it
were better my mother had not borne me: I am very
proud, revengeful, ambitious, with more offences at
my beck than I have thoughts to put them in,
imagination to give them shape, or time to act them
in. What should such fellows as I do crawling
between earth and heaven? We are arrant knaves,
all; believe none of us. Go thy ways to a nunnery.
This diatribe is directed at Hamlet's mother more than Ophelia. Remember, Hamlet is more angry at his mother's unnatural relationship than avenging his father's murder. She's sleeping with the enemy, and he wants to punish her, if not physically then verbally--if not directly, then indirectly through Ophelia. Remember, the Ghost told Hamlet to leave Gertrude to heaven, so Hamlet must work through Ophelia to avenge his mother's sins. So, Hamlet projects his Oedipal jealousy and anger onto Ophelia, who is a symbol of female corruption like his mother.
I think there are several reasons for his behavior. The most powerful, in my mind, is the real love he feels for her and the confusion and heartache he feels because she has refused to see him but hasn't presented a real reason for the change of heart. If we look to his emotional outpouring at her grave, it is clear that he really felt strongly for her, it wasn't just a dalliance as Polonius thought it was at first. So this fact that she is returning the things he gave to her with such feeling is infuriating.
Add to that his immense frustration at his own inability to be decisive and to act on what he thinks is his dear father's murder, and the fact that he feels he is being spyed upon and attended most dreadfully and his anger boils to the surface as he takes some of it out on this woman he loved and who loved him back but now cannot seem to reciprocate for reasons unknown.