I think there might have been some sincere anger toward Ophelia for not being someone he could truly confide in. He knew that she was an obedient daughter and that anything he said to her would be reported to Polonius (whether that is true or not, I don't know - right now I'm only talking about Hamlet's perceptions).
I also believe he was really frustrated and angry with his mother's willingness to so quickly remarry, which became anger toward all females for what he considered their lack of loyalty and faithfulness:
"Heaven and earth,
Must I remember? Why, she would hang on him
As if increase of appetite had grown
By what it fed on, and yet, within a month—
Let me not think on't—Frailty, thy name is woman!—" (1.2)
Anger toward Gertrude was taken out on Ophelia because there was only so much he could do to his mother.
I truly believe that Hamlet knows he will not survive the confrontation with Claudius over his murdering his father and marrying Gertrude to seize a title that does not belong to him. In being cruel to Ophelia, Hamlet believes he is protecting her from emotional and physical harm. He does not, however, anticipate her suicide. The death of her father and absence of her brother in addition to Hamlet's rejections are too much for the young woman to handle.
Had she been stronger mentally and emotionally, I would say that Hamlet did not go overboard in his treatment of her.