We need to think of the way in which Joyce uses such details to paint a picture of the setting and to describe the kind of life of the narrator and of those around him. The story is set in Ireland, a strongly Catholic nation, and this is a central part of Irish identity. In particular, in this story, religion is something that seems to be associated with the general sense of drab reality and decay that surrounds our narrator. Consider how the priest is mentioned at the beginning of the story:
The former tenant of the house, a priest, had died in the back drawingroom. Air, musty from having been long enclosed, hung in all the rooms, and the waste room behind the kitchen was littered with old useless papers.
Death, decay and drab reality seem to be the environment in which this young narrator grows up and seem to be the antitode to his own youthful raptures when he imagines himself to be in love with Mangan's sister. The mention of the dead priest therefore is symptomatic of the entire setting or urban Ireland and the impact that it has on the narrator. A very drab scene is described to us, and an essential part of this reality is Catholicism.