If you have a look at the conversation between Charles and Oliver, you can see what has happened at the court that explains why the old Duke has fled to the forest with some of his lords. His power was usurped by his younger brother, and so Duke Senior, with some of his faithful lords, fled to the Forest of Arden, thereby forfeiting their property and wealth, and thus are left to wander around unmolested. Note what Charles says specifically in response to Oliver's question about the news in court:
There's no news at the court, sir, but the old news: that is, the old Duke is banished by his younger brother the new Duke, and three or four loving lords have put themselves into voluntary exile with him, whose lands and revenues enrich the new Duke; therefore he gives them good leave to wander.
Thus we see that the old Duke Senior left the court after a struggle for power that meant he was forced to flee and see safety in the Forest of Arden. The theme of fraternal conflict is thus reinforced. We have seen the conflict between Oliver and Orlando, and now we hear of two other brothers who are in a state of conflict.