The lines you are interested in read in full,
Yea, from the table of my memory
I'll wipe away all trivial fond records,
All saws of books, all forms, all pressures past,
That youth and observation copied there;
And thy commandment all alone shall live
Within the book and volume of my brain,
Unmix'd with baser matter: yes, by heaven!
Hamlet is speaking to himself (a soliloquy) after the Ghost of his father has told him of the treachery of both his uncle and mother. He vows to forget his intellectual pursuits, and everything that he previously held dear in his life (like his mother's love) and instead will replace all of those things with a singleminded pursuit of revenge and justice for his father.
Hamlet uses metaphor, ie, "the table of my brain" to help him envision everything in his life that has transpired previously, thinking of his books and experiences as being laid out for him to see, and all is meaningless clutter compared to that which now dominates his life: betrayal, murder, and revenge.