Hamlet senses that they are mimetic. That is, they are feigning concern for him and play act as they speak to Hamlet, replying in non-commital phrases such as "Happy in that we are not over-happy" (II.ii.221). When Hamlet inquires of them the purpose of their visit, Guildenstern asks, "What shoud we say, my lord?" suggesting again their acting. Hamlet replies,
Anything but to th' purpose. You were sent for, and there is a kind of confession in your looks, which your modesties have not craft enough to color....(II.ii.265)
Hamlet realizes that his two friends are not being true to him; instead, they are acting under the pretense of concern for him in order to learn what they can about Hamlet's feelings and thoughts so that they can report to the king and queen.
(Links to two other questions on Guilderstern and Rosencrantz are listed below)