As far as I can tell, there is no good reason for Touchstone to say this about Sir Oliver Martext. However, it is not surprising that he says it anyway. I think the reason why he says this is to be found in what kind of a character he is.
Remember that Touchstone is a clown and that his role in this play (it seems to me) is just to be annoying and funny and to serve as a contrast to the other characters. So I think he is just saying this to contradict Audrey. He may also be referring to the idea that marriage is a trap for men and saying that Sir Oliver was wicked for trying to have him marry Audrey.