At this point in the play Hamlet has just learned from the ghost of his dead father that it was Claudius who poisoned him and took the throne and his wife. He has commanded Hamlet to avenge his "poor father's death." But Hamlet knows that he must be a bit cautious. Before he can take such rash actions against the current King, he must investigate the charges and make sure that Claudius is, in fact, guilty of the crime. His first plan of action is to "put an antic disposition on." He plans to act crazy around the court and castle in an apparent attempt to catch Claudius with his guard lowered in a position where he might reveal the truth of what he did. Hamlet hopes that by acting crazy he won't be regarded as a threat, so people may speak more freely around him than they might otherwise.
In lines 165 and following, he tells his friends HOW is going to be acting so that they are not overly worried about this upcoming strange "change" in Hamlet's behavior. He specifically says that he is going to do some of the following: "his arm encumbered thus (this way), or this head-shake / Or by pronouncing of some doubtful phrase." He is going to walk around acting weird! He doesn't want his friends to be alarmed, but he also doesn't want them to reveal that they know the truth, so he asking them to swear that they will keep secret everything they have seen and heard this strange night.