How do Rosencrantz and Guildenstern decide what to tell Claudius about their encounter with Hamlet?
Rosencrantz and Guildenstern do not necessarily leave things out; they are just given vague orders and reply with vague responses. The reveal to Hamlet that they'd been sent to see what was troubling him after he provokes it out of them. And they report back to Claudius that Hamlet is distracted but his spirits were momentarily lifted when he heard of the proximity of the players, wherein he would use a play to “catch the conscience of the king.”
Rosencrantz and Guildenstern are in a unique position here; almost as if they are not in the play. They relay information between characters, keeping it as vague as possible in order to literally not affect anything as far as character motivations or plot. This is underscored by Rosencrantz's reply to Hamlet that they are “indifferent children of the earth” (II.ii.1329). They merely confirm characters' thoughts: Claudius thinks Hamlet is up to something and vice versa.
When Claudius asks them to check up on Hamlet and cheer him up, they ask to be commanded to do so. This tells me that they don't want to be duplicitous with their friend, but they will go along with it if commanded by the King to do so. That being said, they still do not really betray Hamlet nor do they fail in following the King's command. They are either brilliantly or ignorantly vague in how they handle the whole thing.
This is a play about playing; a play about acting in different ways in order to affect the way others act. You could say that Claudius attempts to use Rosencrantz and Guildenstern the same way Hamlet uses the Players. It would then seem that, between Hamlet and Claudius, Hamlet is the better director. You could also say Hamlet is just plain smarter, being that he used professional actors and Claudius used Hamlet's closest friends. Hamlet would, and did, surely know when they, his closest friends, are 'playing.' He was not fooled. Claudius, on the other hand, was exposed by the Mousetrap.
In response to your questions, I don't think they are treated unfairly by Hamlet; by Claudius, maybe. Hamlet reveals little to them; he is well aware of their orders, replying to Guildenstern “you cannot play upon me” (II.ii.2249). They are like the players, pawns. I also go back and forth on whether they are ignorant or brilliant in the way they remain so vague. If anything is left out in their report to Claudius, it is the line where Hamlet says, “I lack advancement” (II.ii.2221). Hamlet thinks that Claudius intends to have someone else succeed to the throne; perhaps to kill Hamlet. But I would argue that Claudius and Hamlet are both aware of eachother's distrust of one another.
At times, it seems impossible to tell when Rosencrantz and Guildenstern are being genuine or 'playing,' blurring the line between life (being) and art (playing). Check out Rosencrantz and Guildenstern are Deadby Tom Stoppard.