I think you are asking about the two possible meanings of the word "nunnery." The first definition is literal; a nunnery is a house for nuns who are living under the holy order of the church in a life of seclusion from men. Hamlet may intend this definition because he immediately explains that Ophelia should go to a nunnery where she will be away from men and therefore be unable to be a "breeder of sinners." He later says that "We [men] are arrant knaves all; believe none of us." He is making a disparaging remark about all men and is warning her away from the foolish liars that all men are. He is clearly venting his frustrations with humanity.
Later in the scene tells her again "get thee to a nunnery," but this time he says it after he tells her that if she marries he hopes that she is "chaste as ice" (frigid, sexually) and then adds another insult by saying the "wise men know well enough what monsters you make of them." He continues his insults by talking about how women put on a false face and flirt and act silly as a means to attract men's attention. Here is where the second defitinion, a slang expression of Shakespeare's age, may come into play. Nunnery was used as an expression to mean brothel or house of prostitution. If Hamlet really means what he says, he is being exceptionally cruel to a woman he supposedly loved. He is rightly angry that she is in on the plot to get information from him to prove something to Polonius and Claudius, and perhaps he is lashing out from that sense of betrayal. We also know that he is especially disgusted by his mother's relationship with Claudius, and that he may be extending or generalizing that disgust to all women, and Ophelia is a convenient victim. Either way, this scene shows Hamlet's crazy act in full force, but as always, he makes some sense even in his crazy act.