International institutions such as the United Nations weaken the nation-state by setting limits on what those nation-states can do.(Actually, it is better to use the term "state" in this context. A nation-state is a state (country) that is made up almost exclusively of people of one "nation," or cultural group. Japan is a nation-state. Many countries are states without being nation-states. The US is a state but not a nation-state.)
When states join international institutions, they agree to limit their freedom of action. When a country joins the World Trade Organization, for example, it agrees to abide by the rules of that organization. If it breaks those rules (for example, by imposing a tariff), it can be sanctioned by the organization. This is not supposed to happen to states. They are supposed to be completely sovereign over everything that happens in their territory. By joining international organizations, they weaken their sovereignty, thus weakening themselves as states.