In the Preface, "The Lives of Victor Klemperer," by Martin Chalmers, reason for why Jews, who were urged to go, did not leave Germany before 1941 are enumerated. Some reasons included age, most who stayed were older than 45; gender, most who stayed were women; the difficulty of removing assets; the difficulty of finding willing host nations; reluctance to abandon other family members, friends, possessions and homes; reluctance to abandon national identity as German; reluctance to abandon careers and professions (p. xv).
Klemperer's "May 31, Friday" and "June 11, Tuesday after Whitsun" entries encapsulate several of these reasons: home, professional work, German identity, friendship. Regarding friendship, Klemperer speaks of having his hand warmly shaken by the beadle at the college when he goes to hand in his key:
A beadle, whom I only know by sight, came he was wearing an SA uniform; he shook my hand with evident warmth .... (p. 125)
He also speaks of being emotionally shaken at the news of the death of a man, Heiss, who was of his own generation. He also speaks of the receipt of a letter after two years silence from his friend, Harzfeld.
Regarding German identity, Klemperer's identification with Germany is illustrated in his emotional response to the outing he took with Eva to scenic Bastei. He is grateful for the beauty and for Eva's ease and enjoyment in getting around in and appreciating the scenery. He expresses this by declaring that they must stay where they are. This incident also illustrates the reason for staying pertaining professions and careers. Klemperer declares they must stay even if a professional opportunity is offered from someplace "outside," such as the University of Constantinople, which made the offer Dember received and accepted.
We must stay here and scrape by, even if something is offered from outside; I cannot imprison Eva.
Regarding home, the visit to Bastei further illustrates this reason for not going away from Germany. After the tour, the party sits by the forest and has tea but before beginning the outing for the day, the party, consisting of Klemperer, Eva and the Isakowitzes, take a tour of Klemperer's home and they all drink coffee there.
After the visit to the Bastei, we rested by the forest, and had tea. Before the excursion we had shown off our little house and drunk coffee there. (p.125)
If by "Victory" you mean "Victor," then I have to say that his biggest reason for not leaving Germany is that he recognized that he had to "bear witness" to what was going on and meticulously record it. However, there are repeated references to in his diaries stating that "emigration was hard" or becoming ore difficult. This seems to suggest that he wanted to leave, but couldn't because the increasing restrictions placed on the Jewish people. Hope that helps.