Why is it so important to Holden that Phoebe not be mad at him? Why is it so important to Holden that Phoebe not be mad at him? Relate this to his past, and Allie.
As Phoebe is still a child, she is yet to be corrupted by the 'phoniness' that Holden sees taints all of the adults around him. He values her viewpoint as truthful, as he did with his brother, Allie. They are the only people, he feels, really understand and care for him and can be honest with him. It is unsurprising that having lost Allie and been away from Phoebe, Holden is heading down a destructive path which leads to his eventual breakdown.
Holden has no real friends, nor does he like many of the people he meets. The only people he values are his sister, Phoebe; and his dead brother, Allie, who Holden treats as if he is still alive. It is important that Holden loves someone, and that someone loves him. Phoebe is that single character, one whose respect Holden desperately wants to maintain.
Phoebe is all that Holden really has left in the world, right? He had Allie, but Allie died. So now Phoebe is the only person in the world that Holden actually trusts and who Holden does not see as a "phony." Given that, it makes a lot of sense that he would care about how she thinks of him.