Oligarchical collectivism refers to the system by which Orwell's dystopian future nations are governed. In some sense it is a paradoxical term, since a "collectivist" society, at least ideally, is one in which the people as a whole, the "collective," govern themselves and control the means of production, instead of a small owner class controlling it as in a capitalist system. Yet an oligarchy is a small elite group that has absolute power. The Soviet Union was ruled by just such an oligarchy, though the Soviets claimed to have created a worker's state in which all people were equal.
Orwell projects the Soviet system into the future and gives it an even more extreme and exaggerated form than what existed in the real world under Stalin. In 1984 the oligarchy is the Inner Party, the group of which O'Brien is a member. These people have unlimited and arbitrary power over the rest of society. The organization of that society is collectivist because nothing is privately owned. The state, which is supposed to represent the people, owns and controls everything.
The irony, or the contradiction, in the system is that the state oppresses the people rather than representing them. It is not even a case of the transitional system Marxist theory envisions as occurring between capitalism and communism. The elites of Oceania, as O'Brien reveals to Winston, are not governing for the good of the masses as they publicly assert. They are interested only in power for themselves as an end in itself. Oligarchical collectivism is a form of socialism, but it's the opposite of the democratic socialism Orwell himself believed in. The people as a whole are disempowered under the pretext of equality for all.
In 1984, "The Book," The Theory and Practice of Oligarchical Collectivism, is purported to have been written by Emmanuel Goldstein, the enemy of the Party. In reality it has been collectively written by O'Brien and others as a blueprint for the world they have created, though its stated intent is to condemn the system by which that world is governed. The members of the oligarchy are thus conscious of their own hypocrisy in claiming to represent the collective while oppressing it to an extent unprecedented in history.
In our own world today, given the fall of the Soviet Union decades ago, only relatively small countries such as North Korea and Cuba can be said to have oligarchical collectivist systems. China is an oligarchy, but it is not truly collectivist, given that private ownership is permitted. Yet even the democratic countries can be considered oligarchical given that corporations and economic elites have disproportionate influence and control over the elected government. The elements of Orwell's dystopia exist in the real world today, though not in the absolute sense shown in 1984.