The answer to this question lies in the comparison between the speaker as he remembers himself thinking and acting as a child and then how he looks back on his childhood and his father's act of love as an adult.
Initially, as a child, the speaker was blind to what his father did. The line "No one ever thanked him" supports this, as nobody ever thought for one minute of what it must have cost his father to get up so early in the intense cold and warm the house up for everybody else. In addition, the speaker remembers descending and then speaking "indifferently" to his father, in spite of the fact that he had polished his shoes as well as warming up the house.
It is only now, as an adult, that the speaker is able to look back and reflect on his father's incredible sacrificial love and how he displayed that through braving the "chronic angers" of the house that he himself feared to brave. This change of attitude is reflected in the last two lines of the poem:
What did I know, what did I know
of love’s austere and lonely offices?The reference in the last line to the "austere" and "lonely offices" of love shows the way in which the speaker has matured and developed. Only with age is he able to look back and recognise how wonderful his father was for demonstrating the "lonely offices" of love.