Why is extreme nationalism a major factor leading to World War I instead of the alliance system?
In general, both extreme nationalism and the system of alliances are seen as major factors that led to World War I. You can see this, for example, in both of the links below. If your instructor says that alliances were not a major factor that led to the war, it is probably because he or she thinks that the alliances did not cause the war but simply made it bigger when it began.
Nationalism clearly helped to start WWI. The event that caused the war to begin was the assassination of Archduke Franz Ferdinand of Austria-Hungary. Austria-Hungary was a multi-national empire that was ruled by ethnic Germans. Many of the people it ruled, such as those in what later became parts of Yugoslavia, were ethnic Slavs. For nationalist reasons, the Slavs did not like being ruled by Germans. It was a Slavic nationalist who assassinated the Archduke in Sarajevo, which was later a part of Yugoslavia and is now the capital of Bosnia and Herzegovina. Austria-Hungary held Serbia (a Slavic country) responsible for the assassination and issued it an ultimatum that led to Austria declaring war on Serbia. This was the start of WWI.
By contrast, you can argue, the alliances did not cause the war to start. The Archduke was killed because of nationalism, not alliances. Austria-Hungary held Serbia responsible because of nationalism, not alliances. Instead, the alliances caused the war to expand once it had started. Once Austria-Hungary declared war on Serbia, Russia joined the war because of its alliance with Serbia and Germany joined because of its alliance with Austria-Hungary. You can argue that this spread the war but did not start it.
Most people would say that both nationalism and alliances were factors that led to WWI, but if you are supposed to argue that nationalism was a factor and alliances were not, this is the argument to make.