To start with, it is important to remember about "Ancestor" is that the father doesn't leave the children for periods of time in a state in which they are uncared for, unprovided for, and unloved. They are with the grandmother who guides them "like a root in the heart of the earth." So during the times of his return, they have not been physically nor emotionally neglected, though it is true he himself did not care for them, they were nonetheless cared for.
The second thing to remeber is that when he returned--and he always returned--he did not return with violence, anger, hostility, judgementalism, or anything negative. He returned with love and words about the children's goodness and value. He gave them the gift of freedom and mind's unclouded by society's false values of judgement and false worth.
Therefore, the love provided by the father was is genuine sincere love recognizing human individuality, intellect, and dignity, a love that gives the gift of an independent spirit. The son understood it in a quiet way because while he yearns for more as he watches his father leave without stirring a twig, as though he'd never been there, and "like a child / with a warrior's heart," he also feels the truth of his father's loving spirit and the affirmation of his own personhood.