The scene this question refers to occurs in Act III scene 2, when Rosencranz and Guildenstern are sent by Gertrude to ask Hamlet to speak to her in her bedchamber following the performance of the special play that Hamlet put on for his mother and uncle. The extended metaphor Hamlet uses arises when the Players enter the stage playing recorders. Hamlet seizes upon this as an apt metaphor to describe how Rosencrantz and Guildenstern wish to "play upon" him as if he were a recorder:
You would play upon me, you would seem to know my stops, you would pluck out the heart of mystery, you would sound me from the lowest note to the top of my compass; and there is much music, excellent voice, in this little organ, yet cannot you make it speak.
Hamlet compares himself to a recorder and states that Rosencrantz and Guildenstern, through trying to manipulate him and make him trust them whilst all the time doing so in order to gain access to his thoughts and feelings for Claudius, are trying to "play" him. The scene ends with Hamlet's defiant statement that he will not be "played" in this particular fashion. In other words, he will not allow himself to be manipulated. It is important to recognise that Hamlet here is not just responding to the attempts of Rosencrantz and Guildenstern to gain his confidence, but he is responding to the cumulative betrayals he has experienced: all of those closest and dearest to him, whom he thought he could rely on, have one by one betrayed him. He has seen his mother and even Ophelia work against him, and his frustration is expressed against Rosencrantz and Guildenstern. Hamlet will not be manipulated any further.