First, let us think about what international norms and institutions are. International norms are the values that have come to be shared in common by the international community as a whole. These norms can be written or unwritten. In their written form, they are embodied in international law and in things like the resolutions put out by international institutions such as the United Nations.
With that in mind, how might terrorism and counterterrorism violate such things? The answer is much clearer for terrorism. There is a clear international norm against the intentional taking of innocent lives. It is accepted that soldiers will kill one another in war. It is also accepted that there will be some accidents and civilians will sometimes die. However, it is clearly understood that civilians should not be intentionally targeted. Terrorism violates this international norm.
Counterterrorism can, but does not necessarily, violate international norms and institutions. It is quite possible to carry out counterterrorism in a way that conforms to international law. However, it is also possible to do things like torturing suspected terrorists. There are international norms and laws about how detainees should be treated. When counterterrorism agents do things like torturing people they capture, they are violating these norms.