'Coyle' is an obsolete spelling of a now obsolete usage of the word 'coil'! In the 16th century, it was a colloquial expression, which fell into literary usage. It means 'confusion', 'din', 'disturbance' or even 'turmoil'; 'to keep a coil' means to make a disturbance.

The jailer's daughter is speaking of her love for Palamon, and so uses the term metaphorically, to mean the turmoil that her feelings create in her heart. It is something over which she has no control, hence she says 'what a coyle he kepes' - making him the active agent.