I like the way your question clearly points towards an understanding of how the speaker is looking back on his childhood from the vantage point of reaching mature adulthood. Thus it is that the poem presents us with two different attitudes and beliefs about the speaker's father. The child that he was obviously took his father's acts of self-sacrificial love for granted. He spoke "indifferently" to his father, and it was clear that he did not know anything of "love's austere and lonely offices."
However, if we think for one moment about how the older and wiser narrator describes what his father did, the respect and love and sense of thankfulness that he has for his father becomes evident. Note the way that he describes the cold as being "blueblack" and stresses the way that his father, even on Sundays, after a week of labour, would get up without fail, even though he was never thanked for this service:
Sundays too my father got up early
and put his clothes on in the blueblack cold,
then with cracked hands that ached
from labour in the weekday weather made
banked fires blaze. No one ever thanked him.
Such details that the speaker as a boy had been blind to clearly indicate the way that the speaker, now he has grown up into an adult, appreciates his father for what he did and recognises the sacrifical acts of love that his father performed, day in and day out, in spite of his own exhaustion, for his son.