The Middle Ages were renowned for their numerous religious feast days, many of which revolved around the veneration of the Virgin Mary. In this pre-Reformation era, Mary was venerated as the mother of Jesus, and many celebrations, feast days, and festivals were held in her honor.
One such feast day, and arguably the most important in the religious calendar was Lady Day, also known as Our Lady's Day or the Feast of the Annunciation. This feast day was held to commemorate the Archangel Gabriel's visit to the Virgin Mary, during which he told her that she would give birth to the son of God.
There is one mention of Lady Day in Adam of the Road. Adam has decided to become a minstrel. Not a minstrel like the de Veseys, but a minstrel like Roger. Although Adam's lost his harp, he's sure that he'll find it again, just like he did the previous two occasions he lost it.
In any case, he's decided to head straight back to London and de Lisle House. As part of his travel plans, he thinks about stopping off to see Jill and John Ferryman. However, Adam soon thinks better of it. He has no time to spare; December's almost half gone and Our Lady's Day is long since past.
As well as being a religious feast-day, Lady Day was New Year's Day, which in medieval England was on March 25. The reference to Lady Day in the story could be seen as a way of drawing attention to how little time Adam has, and why therefore he doesn't want to stop off and stay with the Ferrymans.