The clue to understanding Duke Frederick's suspicions and banishment of Rosalind lies in the preceding scene, Act I, scene ii. The last half of the scene is devoted to the wrestling match between Orlando and the challenger Charles. Several things are revealed at this wrestling match. First of all, it is important to bear in mind that Duke Frederick is there and standing near to Celia and Rosalind since he directs remarks to them. It stands to reason that he can also readily hear what they say and wittiness what they do.
Let's start with what is revealed because of Rosalind and Celia. At the Duke's bidding, Celia and Rosalind attempt to dissuade Orlando from wrestling. His responses make the girls admire his courage and manliness. The result is that they cheer him on. When he wins, Rosalind's emotions go a step beyond admiration and turn to the beginnings of love. This is attested by her gift to him of her necklace.
Giving him a chain from her neck
Wear this for me, one out of suits with fortune,
That could give more, but that her hand lacks means.
With the Duke's vantage point, he sees all and notices that Rosalind is poised to fall in love with Orlando. This knowledge will be important following other developments.
Next, let's look at what is revealed after Orlando wins the match against Charles. The Duke is not displeased that Orlando has won but is greatly displeased to find out that Orlando is the son of Sir Rowland de Boys, because de Boys was an enemy of Frederick:
I would thou hadst been son to some man else:
The world esteem'd thy father honourable,
But I did find him still mine enemy:
Thou shouldst have better pleased me with this deed,
Hadst thou descended from another house.
But fare thee well; thou art a gallant youth:
I would thou hadst told me of another father.
With these revelations in mind, it suddenly becomes clear why Duke Frederick is suspicious and banishes Rosalind in Act I, scene iii, accusing her of being a "traitor" and sarcastically saying all traitors are "as innocent as grace itself" when what he means is that their innocence is a falsehood, a disguise for their treachery. We now see that Frederick is suspicious of Rosalind because she is the daughter of his enemy--the brother whom he deposed from the throne--and she is potentially poised for an alliance of marriage with the son of another enemy, Rowland de Boys. Duke Frederick fears an alliance of his enemies who might plot his overthrow and the restitution of the crown to the rightful wearer, Duke Senior, who is Rosalind's father and Orlando's father's friend. Of course, Frederick would not want to say this and thereby possibly put treacherous ideas in Rosalind's thoughts.