How has the theme of transgression and gender been handled in the 16th and the 17th century European comedy in the genre of drama?
By "transgression and gender", I mean "women's breaking free from gender hierarchy in a patriarchal set-up."
First note the novel had not emerged as a concrete genre during the period which you've specified. Cervantes' Don Quixote (1605, 1615) is sometimes credited as the first novel, but was also considered a "chivalric romance". Daniel Defoe's Robinson Crusoe is often considered the first novel in English, but this was not published until 1719. This doesn't mean that it wouldn't be a fruitful study to find early "novel" examples and explore their gender representations, but drama was the more popular form during this period.
During this period, playwrights produced increasingly gutsy and positive female characters who are able to wield words and intelligence to mitigate the strictures of living in a patriarchal world. However, even these more assertive female characters are often neutralised or "tamed" into marriage, as in Shakespeare's famous The Taming of the Shrew (1591). Beatrice from Shakespeare's Much Ado About Nothing(1599) is another example of a female who refuses to succumb to the male power around her and who more than matches the men in her command of wit. She launches into the play in the first scene in a battle of words with her former lover, Benedick, declaring,
I had rather hear my dog bark at a crow than a man swear he loves me.
At the end of the play, however, she marries the man she had originally scorned, which arguably represents her assimilation into the patriarchal order.
Later in this period, after the Restoration of Charles II to his throne in England in 1660, we see the feisty revival of theatre which had been banned during Oliver Cromwell's Commonwealth. Around this time in France, Moliere became a notable comedic playwright whose presentations of women could be derogatory. In his play, Les Precieuses Ridicules (The Affected Ladies) (1659), he satirises a pair of country girls whose over-confidence causes them problems when they try to join sophisticated city society.
However, the English Aphra Behn, who is often recognised as the first female playwright, began to produce much more promising female characters such as the clever Hellena in The Rover (1677) who manages to outwit the rakish Willmore, an immoral and flirtacious cavalier. Aphra Behn also shines a critical light on rape in this drama, calling into question the attitudes of the male-centric, post-Puritan and morally-carefree times she was writing in. Aphra Behn is a great place to start when considering early feminism in European drama.