Hamlet has forced his mother to listen to him, and has, with his words, held a mirror up to her so that she will acknowledge what she has done in marrying her dead husband's brother. He refuses to let her be until he has offered her a glimpse of her soul, "a glass/ where you may see the innermost part of you". He forces her to acknowledge that she has no good (virtuous/ respectable) reason for her marriage, and in doing this, makes her realize her guilt. She says, "thou turnest mine eyes into my very soul; and there I see such black and grained spots/ As will not leave their tinct." She says he has made her look deep into her soul and what she sees there is black, permanently stained. This suggests that instead of virtue in her soul, she sees vice.
He tells her that she must repent and leave her new husband. This is when she says he has cleft her heart in twain. This timing is significant because we wonder what it is that makes her so devastated--is it her guilt at what she's done, or her desire to continue her relations with her new husband that makes her feel this way?
By this she literally means that her heart is cut in two. But it is unclear as to where the two parts are. Is it between love for her first husband (by all accounts the better man) and her second? Is it between right and wrong? Or is her heart not divided, but rather broken, and she is heartbroken because she feels so guilty? Hamlet speaks of virtue and tells her to throw away the "worser part" of her heart and life "the purer with the other half," suggesting, perhaps naively, that she has hope for some redemption now that he has saved her.