The question you need to think about is are you talking about the son as an adult looking back on his childhood and his father or the son when he was going through his childhood? The poem makes clear that there is a massive contrast between these two different states.
As a child, the speaker is clearly ungrateful and unaware of what his father does for him, taking it for granted and not thanking him. The narrator says that he spoke "indifferently" to his father when he came downstairs after his father had risen so early and warmed the rooms, and even polished his shoes as well. The first stanza states that "No one ever thanked him" for such labours and evidence of sacrificial love.
However, the change in the narrator as an adult looking back at his childhood now is evident through the last two lines when he asks himself the following question:
What did I know, what did I know of love’s austere and lonely offices?The wording of the "lonely offices" of love shows that now the speaker is able to look back upon himself as a child and berate himself for not recognising his father's sacrificial actions and love towards him.
I would argue that the feelings between father and son were ruled by a contrast in what it means to love. As the previous educator mentions, the adult narrator is able to understand that the father's dutiful attention was a demonstration of love, though nothing else in the poem indicates any affection between father and son.
The following lines are quite telling: "When the rooms were warm, he’d call, / and slowly I would rise and dress, / fearing the chronic angers of that house..." There is an emotive contrast between "warm" and "angers." This suggests that, though the narrator's father was dutiful—ensuring that the house was warm, calling his son to rise in the morning, and polishing his shoes—he was frequently angry or frustrated. In the next line, we learn that the narrator spoke "indifferently to him," which can be attributed both to the ingratitude of a youth and the possible need to steel himself against his father's "chronic angers."
In the final line, he is critical of his former behavior: "What did I know, what did I know / of love's austere and lonely offices?" "Austere" describes the father's dutifulness, which did not immediately translate as love. Hayden manipulates the meaning of "offices" to describe the father's chores as well as to emphasize the solitary nature of this unappreciated work. Father and son occupied different spaces—that of caretaker and the one receiving care.