What factors caused instability and change in the Balance of Power System?
The major factor that caused instability and change in the balance of power system in Europe in the late 1800s was the rise of alliances.
Balance of power systems are based on the idea that all essential actors will remain as free agents. All states see all other states as potential allies. They are all willing to make alliances with any other state or combination of states. This is done when a state or coalition is threatening to upset the balance of power.
In the European balance of power system, this fluidity broke down. States started to enter into permanent alliances with and against one another. When this happened, the balance of power system broke down and Europe started to partition itself into set camps whose members were permanently allied with one another and permanently antagonistic to the other side.