Holden calls his brother D.B. a prostitute because he believes that he is using his talent as a writer inappropriately because instead of writing serious books, he writes scripts for Hollywood. Holden believes that D.B. is a sellout, giving up on the intellectual pursuit that he should have as a writer. Now, he writes purely to make money.
The comparison is made with a prostitute, because he/she works only for money, someone who will do anything for money. Therefore, D.B. in Holden's eyes has become a prostitute because he is willing to write anything for money. It is all about the money for D.B. he has bought a new Jaguar, a fancy, expensive car.
Holden has lost respect for his older brother and now groups him in with all the other adult phonies that populate his life. Holden resents his brother's desire to make money, yet it is in this very area that Holden is a true phony. He has money in his pocket put there by his grandmother, who sends him money four times a year. Holden disrespects his grandmother suggesting that she is crazy because she sends him "birthday money" four times a year.
Holden is sadly lacking in his understanding of adult responsibility, real life, making a living, paying bills, needing to make money to pay bills. Holden, in this regard, is very immature and childish. His resentment towards his brother is misplaced and misinformed.
He means that his brother is a sell out. More or so that D.B does not value himself enough and is finding a cheap/fast way of making money instead of finding a "real" job and making his way through. He believes his brother is just writing mediocre stories just for the sake of the money, instead of actually putting time and talent into it.
DB is Holden's brother who started out as a small writer and became a Hollywood screenplay writer. Holden says his brother "sold out" to Hollywood. He felt that his brother's stories were better before he sold them to Hollywood. Holden is the ultimate hipster in this scenario and that's why he calls him a prostitute.