How can a legal system be designed so that it protects the rights of the minority from an encroaching majority?
It is not actually possible to create a legal system that protects the rights of minorities in the manner that you suggest. At least, it is not possible outside of a dictatorship. The reason for this is that strongly held beliefs among a significant majority of the people will override legal concepts of protecting the minority in a democratic society.
We can see in the history of the US that this is true. Truly unpopular minorities have often been oppressed even when the Bill of Rights or other parts of the Constitution forbid this. The most obvious example of this is the fact that blacks were denied their rights between the end of Reconstruction and the mid-1960s.
To give minorities the best chance at retaining rights, there should be a system like that of the US in which rights are explicitly listed in a constitution. There must also, as in the US, be an independent judiciary that has the power to declare that minority rights must (by law) be protected. This is the best chance that minorities have, but it is not foolproof.
I think that any legal system that wishes to protect the rights of the individual against the majority must make this an explicited and stated purpose of its code of conduct. For example, the United States Constitution includes specific amendments that are applicable to all people, all individuals and these help to protect the encroachment of the majority. For example, in the United States Constitution, there are specific amendments that protect the rights of accused individuals in legal cases. While the majority of people may wish to fore go a trial and convict individuals in the court of public opinion, the presumption of innocence is something that applies to all persons accused of a crime. There are specific barriers that the prosecution, or the majority, must meet in order to convict an individual. Even so, individuals found guilty may appeal their verdict or sentences through the appellate portion of the legal code. In the end, the rights that are prescribed in the United States Constitution serve as a shield or a type of check against encroachment of the majority.