Here's the first four stanzas of "The Ballad of Father Gilligan," by Yeats.  They include the details you refer to:

The old priest Peter Gilligan
Was weary night and day;
For half his flock were in their beds,
Or under green sods lay.

Once, while he nodded on a chair,
At the moth-hour of eve,
Another poor man sent for him,
And he began to grieve.

‘I have no rest, nor joy, nor peace,
For people die and die’;
And after cried he, ‘God forgive!
My body spake, not I!’

He knelt, and leaning on the chair
He prayed and fell asleep;
And the moth-hour went from the fields,
And stars began to peep.

I've emboldened the lines that include what the priest sees as his grievous sin.  Actually, his reaction is probably quite natural.  Members of his flock are dying faster than he can deliver the Last Rites to them.  He is dozing off in his chair when he gets summoned by yet still another dying parishioner.  He is overworked and exhausted, but when he reacts to the summons in a very human way, he sees his words and behavior as not fitting for a priest.

God, in the poem, though, seems to understand.  While the priest is praying, begging for forgiveness, he falls asleep and misses the man's dying moments, but God sends an angel in the priest's place to administer the Last Rites.

In this poem, the main character, Father Peter Gilligan, asks God to forgive him because he has spoken words that are unworthy of a priest.

When Father Gilligan heard that someone else needed his help, he complained that he never has any peace because people are always dying and needing him.  This is unworthy of a priest because priests are supposed to be called by God to take care of people who need spiritual help, especially as they are dying.