The way that Holden rejects becoming part of society and adulthood is by creating a set of personal rules, axioms, and ideals that represent his resentment against both. These ideals and rules are also ways for Holden to protect himself from possible loss and/or rejection of his peers.
What Holden resents the most is that he cannot find his place in an ever-changing society. He cannot find his place within his own growing process, either. He feels that he is basically falling behind with both. By making his own rules, and ideas, Holden can choose who is "good" and who is "bad". In his case, anyone who does not abide by what he considers to be "real" is immediately labeled as a "phony". Hence, out of that rule that he made for himself, he can now classify his brother, some peers, and some other adults as "phony" as well. Moreover, he can also choose not to mingle with them and use their phoniness (and not his anti-social tendencies) as an excuse for not communicating.
He also makes the exception that children are to be respected, loved, and always protected. This is what gives him the idea of wanting to be a catcher in the rye that would protect the kids that may fall over. This specific life rule Holden makes in honor of his younger brother, Allie, who died of leukemia. His death turned Holden's life upside down. By projecting his love for Allie onto a wish for protecting children, Holden has the opportunity to grief.
Moreover, Holden also makes a rule to only be kind and speak respectfully to his younger sister. To him, she is the only female worth treating nicely. Since he is unable to connect with his peers, he finds protection and attention in a younger child. This is another way in which Holden tries to find a role to fulfill under the protection of his own little rules.
Therefore, Holden cocoons himself inside his own world of beliefs and assumptions by distancing himself from a changing society, and by abiding by rules that he makes for himself in lieu of facing his inner fear of rejection.