As far as I can tell, there is no good reason for Touchstone to say this about Sir Oliver Martext.  However, it is not surprising that he says it anyway.  I think the reason why he says this is to be found in what kind of a character he is.

Remember that Touchstone is a clown and that his role in this play (it seems to me) is just to be annoying and funny and to serve as a contrast to the other characters.  So I think he is just saying this to contradict Audrey.  He may also be referring to the idea that marriage is a trap for men and saying that Sir Oliver was wicked for trying to have him marry Audrey.