The specific information you need to answer your question begins in Act 3.4.151 of Shakespeare's Hamlet:
...Confess yourself to heaven.
Repent what's past. Avoid what is to come.
And do not spread the compost on the weeds
To make them ranker.
Hamlet wants Gertrude to repent having married the brother of her dead husband, and avoid being with him in the future. Hamlet is more specific later:
Good night--but go not to mine uncle's bed.
Assume a virtue if you have it not. (Act 3.4.161-162)
Don't sleep with Claudius, he orders. Pretend to be virtuous even if you are not.
Not this, by no means, that I bid you do--
Let the bloat king tempt you again to be,
Pinch wanton on your cheek, call you his mouse,
And let him, for a pair of reechy kisses
Or paddling in your neck with his damned fingers,
Make you to ravel all this matter out:
That I essentially am not in madness
But mad in craft. (Act 3.4.185-192)
Don't let the king seduce you into telling him that Hamlet is really not mad, but only pretending to be.
Basically, then, Hamlet tells Gertrude to repent, to not sleep with Claudius anymore, and to not reveal to Claudius that Hamlet is not really mad.
Incidentally, that is exactly what Gertrude does. When she next meets Claudius and he asks her about Hamlet, she replies that Hamlet is:
Mad as the sea and wind when both contend
Which is the mightier. (Act 4.1.7-8)
The problem is, though, that the reader isn't sure if she is saying this to protect Hamlet as Hamlet requested, or if she is saying this because she believes it. As is quite common in Shakespeare, the situation and the dialogue are ambiguous.
What Hamlet wants his mother to do is to stop being married to Claudius. He thinks it is disgusting that she has married her dead husband's brother. He also thinks that she was already sleeping with Claudius while her first husband was still alive.
So he would like her to stop being married to Claudius. If not that, he wants her to at least stop sleeping with him. He thinks that it is inappropriate and even incestuous for the two of them to be sleeping together.
He thinks that if she stops sleeping with Claudius, she will be living a purer life.