In Darkness at Noon, the first term, anti-vivisection morality, is used during Rubashev's interrogation by Ivanov. The Party operative tells Rubashev that actions should be based on the need to sacrifice human lives for the good of the greater number, the collective. He dismisses Rubashev's references to Dostoevsky's Crime and Punishment as nonsense and claims that there are only two ways of looking at human life. The first, from the Christian perspective, which he terms "anti-vivisection morality," meaning that any individual life is sacred; and the second, "vivisection morality," meaning that it is "moral" to dispose of human lives if by doing so one can either save a greater number of lives or accomplish a greater purpose. The latter, of course, is the Party's "morality," and what it comes down to is the principle that the end justifies the means.

The second term in your question, the grammatical fiction, is formulated by Rubashev when he considers that the Party discourages its members to speak in the first-person singular. Rubashev has innumerable silent monologues throughout the story, during which he realizes that a monologue is actually a dialogue with a "silent partner." But this mute being is actually what we would (though Koestler does not) more conventionally term our "conscience" or "alter-ego." It is I, or me, but the "shyness," induced by the Party about the use of those pronouns makes Rubashev decide to call this silent partner "the grammatical fiction."

All of this needs to be understood in the context of the totalitarian society of which Rubashev, until being discredited and arrested, has been a leading member. The novel is a fictionalized account of the Stalinist purges of the late 1930s in which men who had been leading Party members were arrested in large numbers, tried and forced to sign false confessions that they had betrayed the state, and were executed. Rubashev is one of these men—one of the "old Bolsheviks" such as Leon Trotsky or Nikolai Bukharin, though in the novel the real names are never used: Stalin becomes simply "Number One" and even the country is nameless, though it's obviously the Soviet Union. The two terms in your question relate, respectively, to the totalitarian disbelief in the value of individual life; and, as a corollary of that, the refusal to see the individual (and the individual's conscience) as a reality. By reducing this entity to a "grammatical fiction," Rubashev himself implicitly accepts the Party's version of reality.

There is obviously a resemblance between this and the disappearance of the pronoun "I" in Ayn Rand's Anthem dystopia, though given the distance between Rand's pro-capitalist views and Koestler's left-wing (despite his criticism of the Soviet Union) orientation, few people would think to connect those two books. Because George Orwell, on the other hand, was a friend of Koestler and had read Darkness at Noon and his other books—and written favorably about them—many see a connection between Darkness at Noon and 1984. Orwell's book is a projection into the future of the regime described by Koestler, and O'Brien's interrogation of Winston is a literary descendant of the "hearings" in which Rubashev is questioned and broken down by the interrogators Ivanov and Gletkin.