According to Max Weber, there are three types of legitimacy. Each is based on a different source. Legitimacy can break down regardless of which source it comes from.
The first type of legitimacy is traditional. In this type of legitimacy, the ruler derives their authority from tradition. They lead because that is how things have always been. This is the sort of authority that the Pope has, or that kings have. They rule because it is traditional that people from their family rule or because it is believed that God has given them the right to do so. This sort of legitimacy can break down if rulers stop following tradition. If they do not act in the ways that tradition says they must, people may stop accepting their authority.
The second type of legitimacy is legal or rational. This sort of legitimacy comes about because the ruler or leader has been selected through legally prescribed methods. This is the type of legitimacy that an American president has. Once a president is elected, the vast majority of people agree that the president has the right to lead the country because he (or someday she) was elected according to the rules. This sort of legitimacy can break down if the leader is seen to violate the rules. Since legitimacy depends on the law, a leader who does not seem to respect the law can lose legitimacy.
Finally, there is charismatic legitimacy. People believe such a leader has the right to rule simply because of the magnitude of their personal talents and charm. This sort of legitimacy breaks down if the leader acts in ways that undermine their charisma. If a leader has legitimacy because she can make people love her, she might lose that legitimacy if she enacts policies that make people stop loving her. Charismatic legitimacy lasts only so long as the leader retains his or her charisma.
Of course, other things can happen to take away legitimacy from a leader. This typically happens when events go badly and the leader does not seem to respond well to them. This can break legitimacy down regardless of what it is based on.