In most areas of the world, there are moths.  A lot of the moths that people see come out around early evening.

So when the poor man sends for Father Gilligan, it is early evening.  Most of the people are asleep because this is before people were staying up late with electricity and such, but it is still not the middle of the night.

We can see this because as Gilligan sleeps, the moth hour ends and the stars come out.  Later on, the moths come again when it gets to be a bit before dawn.  That is when Gilligan wakes up.