The "closet scene" is the turning point in the play. This scene reveals Hamlet's Oedipus complex. Hamlet is in her closet, beside her bed. To have a bed on stage in a Shakespeare play is a big deal. It only happens in Othello and Hamlet, and it was the first time it had ever happened. And to have a mother and son and--in Othello--a black man and white woman around a bed is scandalous for the time. The implication and threat of sex is palpable.
Hamlet is not insane; he is jealous of Claudius (the father). Hamlet feels Gertrude should devote herself to him, her grieving son, instead of Claudius. He wants to kill Claudius, not because the Ghost tells him to, but because he has psychic hostility toward the same-sex parent. Hamlet wants to take mortal revenge on her, I think, but cannot. The Ghost appears and reminds him again to leave her to heaven. So, Hamlet takes verbal and emotional revenge on Gertrude. He unleashes an ad hominem attack of her with justification: she has been sleeping with the enemy.
Most importantly, Hamlet kills for the first time. He obviously thinks it's Claudius behind the curtain, but it shows that he has, he thinks, overcome his inability to actively take vengeance. He wants to take vengeance on Claudius, but he has been too heavily guarded. Now in his mother's closet, to have the King spying on him gives him justification for killing.
Killing Polonius sets up Hamlet's death in Act V, as it gives motivation for Leartes to kill him. Misdirected revenge begets misdirected revenge. Claudius has used Gertrude, Ophelia, and now Polonius to do his dirty work for him, and all three of these pawns will be killed off as a result.
This whole scene is taken up with this confrontation, so this answer will really just summarize very briefly.
Hamlet goes to his mother's room because she has summoned him so she can scold him for the way he's been acting.
But instead of getting scolded, he starts attacking her verbally. He essentially spends much of the scene telling her how much better his father was than Claudius and denouncing her for taking Claudius as her husband. He also implies that she had a part in or at least knew about the murder of his father.
It is also during this scene that Hamlet kills Polonius.
He does not appear to be insane at this point. Gertrude thinks he is because he talks to the ghost, which only he can see.