This quote is from Act I, Scene 5. Hamlet has just learned from the ghost of his father that Claudius has murdered the former king. He has not been pleased by his uncle's marriage to his mother so soon after his father's death, but at this point he becomes outraged. Essentially, this simply means that Claudius, who is gregarious and smiling, not to mention seemingly earnest in seeking his new stepson's approval, is a villain. Hamlet is saying that you can't trust anyone: The ones who come to you with smiles on their faces might be villains. The quote takes on added significance later in the play, when Hamlet's old friends Guildenstern and Rosencrantz, who also come to him with smiles, are in collusion with the king.
There are several implications in Hamlet's observation that one may smile and smile and be a villain. One is to characterize King Claudius. The actor who plays the role should understand that Shakespeare intends this character to be the sort of cunning and unscrupulous person who is always smiling. There are plenty of such people around today. They smile to disarm others. Their real characters are the exact opposite of what they pretend to be--i.e., friendly, warm, sympathetic, kindly, etc. We should caution ourselves to beware of people who are always smiling, especially to beware of strangers who approach us with a smile. What do they want from us? Malcolm in Macbeth tells his brother, "There's daggers in men's smiles." It is especially appropriate that Hamlet should want to write down his insight that a person can smile and be a villain because the play is largely about a young man who leaves school and begins to get a real education into what the world is really like. A modern parallel to Hamlet is the excellent movie The Graduate starring Dustin Hoffman. There are several scenes in Hamlet in which the actor playing Claudius should be calm, soft-spoken, polite, and always smiling. For instance when Hamlet is brought before him after killing Polonius, Claudius should not thunder at him but should give him a warm, confidential smile when he says, "Now, Hamlet, where's Polonius?" And Claudius should certainly be calm and smiling when he is confronted by Laertes leading an angry mob. Hamlet hates that smile. He sees it as a sort of victory smile. ("I've won, you've lost.") And Hamlet knows it is completely insincere. He dislikes Claudius and doesn't understand his uncle's character until his father's ghost tells him about the murder in the garden. Then everything becomes clear in a flash.