In the first scene of the play, Egeus does not explicitly say that Lysander is not worthy. However, we can look at his words and infer what he is thinking.
From his words, it appears that Egeus thinks that Lysander is trying to fool Hermia. Egeus seems to think that Lysander does not really love her but is just trying to seduce her. In his first speech, Egeus accuses Lysander of this. He says
Thou hast by moonlight at her window sung,
With feigning voice, verses of feigning love,
And stolen the impression of her fantasy
With bracelets of thy hair, rings, gawds, conceits,
Knacks, trifles, nosegays, sweetmeats, messengers(35)
Of strong prevailment in unharden'd youth;
With cunning hast thou filch'd my daughter's heart
This accusation implies that Egeus thinks Lysander is unworthy because he is not sincere in his affections.
Egeus also has one other thing against Lysander. This is the fact that Lysander is challenging Egeus's authority over his daughter. Lysander is trying to get Hermia to go against what her father has ordered. This is why Egeus says that Lysander has
Turn'd her obedience, which is due to me,
To stubborn harshness...
By saying these things, Egeus is saying that Lysander is unworthy because he does not truly love Hermia and because he is trying to undermine Egeus's authority as a father. Egeus thinks such a man should not be allowed to marry Hermia.