In the first stanza, the speaker seems to be an adult, thinking back on his childhood. He describes his father getting up early to warm the house before others awoke. The father did this alone, and although his family must have known he did this, the stanza ends with, "No one ever thanked him." That he got up on Sunday "too" implies he did this on other days, perhaps every day. And he got up in the "blueblack cold." This obviously indicates that it was cold but could also indicate that it was still dark outside. The imagery utilizes two senses, touch and sight, to communicate the cold and darkness of the early morning.
The speaker, as a child, wakes to "hear the cold splintering," the embers of the fire perhaps. The imagery uses sound to convey a sense of the heating house breaking apart the coldness. Again, touch is also implied because the house has been heated.
In the last stanza, the speaker recalls talking without care, "indifferently" to his father - even though his father had sacrificed his mornings to heat the house and polish the boy's shoes. As an adult, in hindsight, he laments the fact that he didn't realize that this sacrifice was done out of familial love.
What did I know, what did I know
of love's austere and lonely offices?
As a child, although all of these senses (touch, hearing, sight, sound) indicated the effects of his father's sacrifice each morning, he never realized their significance. He was physically aware of these effects but never (mentally) considered their importance. The "chronic angers" might shed some light on what could have been a strained family dynamic but there is no indication in the poem of what those angers are exactly.