When Holden makes this statement, he means that once you get close to someone and depend on them for emotional support and you have a real connection, you can really miss that person, because they can leave you. People leave, they die, they are unreliable.
Holden technically has lost both his brothers. Allie, his younger brother died, and D.B. has left and gone to California. He feels very distant from D.B., because he thinks he is a sell-out, writing movie scripts instead of serious books. Two people Holden really needs in his life are gone.
Jane Gallagher, the girl that Holden really cares about, she seems far away to him. He misses her too, even though all he has to do is call her. He can't or he won't, and because of this inability to get close to someone, other than his sister, he suffers from isolation.
He fears sharing his feelings with anyone; he has developed a protection strategy, don't get close to anyone and you are safe. He is seriously misguided in this belief, and because of it, he has an emotional breakdown.