As can be seen in the link below, self-determination is generally defined as the right of a national majority (usually defined in ethnic terms) to choose whether it wishes to rule itself as an independent state, to be part of a federal system within a larger state, to be fully assimilated into a larger state, or to occupy some status in between those options. This was one of the major ideas that Woodrow Wilson pushed for in his 14 Points that were offered at the Paris Peace Conference. Wilson wanted all ethnic groups (or at least white ones) to be able to choose their own political status.
The question of what the limits of this should be is a very difficult one. We see it crop up in places like Sudan where a region that was formerly part of the country has broken away to form a new state, South Sudan. It is very complicated because national groups have in many places been split up and have members in multiple countries. An example of this is the Kurds who live in significant numbers in Iraq, Iran, Turkey, and Syria.
My own take on this is that national groups that have been part of a larger state for a long time should not break away and become independent. This would, in my mind, be too disruptive of the international order because it would potentially lead to ethnic groups splitting off into smaller and smaller states around the world. However, I would also argue that ethnic groups who are being abused by the countries in which they live should be given the chance to break away.
This is a very complex issue with no easy answers. Overall, I am in favor of ethnic groups learning to live together within larger states, but I realize that this is not always feasible. My answer to the question would be that there can be no hard and fast limits. Self-determination should be allowed mostly in cases where the ethnic minority has been historically abused.