Hamlet has lost faith in humanity at the point when he meets Ophelia in this scene. His uncle has allegedly murdered his father. His mother has married his uncle. His old friends have come, supposedly to see him, but he has already figured out they were "sent for" by Claudius to scope out Hamlet's state of mind. And now, here comes Ophelia, the girl he is deeply in love with, to give him back the letters he has given her.
The dialogue also implies that Hamlet knows he's being listened to, and if he has figured that out, by the time he utters the line "Get thee to a nunnery" toward the end of the scene, he probably suspects that Ophelia has taken part in the plot against him.
Everybody looks pretty bad from Hamlet's perspective. So why should Ophelia get married and have children, or, in other words, make more people like Claudius, Polonius, his mother, Rosencrantz, Guildenstern, and Ophelia?
Hamlet wants her to shut herself off from the rest of the world—which, in a convent, she would have to do. However, you could argue he still loves her and is saying this out of jealousy: if she's a nun, no other guy will be able to pursue her. Either way, he wants her to be punished for breaking his heart and being the tool of Claudius.
On a small side note, the word "convent" or "nunnery," in Elizabethan England, was a slang term for a house of prostitution. So, Hamlet may be calling her a rude name for breaking up with him (while at the same time expressing a deep-seated misanthropy).