No, I do not believe that Adoniram neglected his responsibilities. He is a husband to Sarah and a father of two children, Nanny and Sammy. His primary responsibility is to provide for his family. He did that, and he is continuing to do that. His family has a house. His family has a barn and owns livestock. Adoniram is financially providing for his family.
Early in the book, it is stated that Adoniram is building a new bigger barn to house more livestock. This will allow him to grow his business and provide more financial security to his family. Also early in the story, Sarah does remind Adoniram that he promised her a bigger house. He has not done that yet, but that doesn't mean he is neglecting his responsibilities. In his opinion a larger barn will better provide for his family. It allows his family to earn more. A larger house will not do that for his family.
About the only way that I can think of in which Adoniram neglects his responsibilities is that he doesn't effectively communicate his intentions with his wife. He doesn't effectively explain to her why a bigger barn is better for the family than a larger house. But if I were to apply that neglected responsibility to Adoniram, I feel that it also must be applied to Sarah. The book ends with Adoniram crying because he finally realizes how important a larger house is to Sarah. Perhaps he never realized it because he didn't listen to his wife closely enough. Perhaps he's too bullheaded and stubborn. Perhaps Sarah didn't effectively explain to Adoniram why a bigger house would provide a benefit to the family equal to a larger barn. Instead she moves the kids and house furnishings into the new barn without her husband's knowledge while he is away on business.
I feel that Adoniram met his responsibilities as a husband and father. He is a good provider to his family. What he and Sarah are guilty of though is poor communication. Neither person voices their thoughts well, nor do they listen well to the needs of the other.