This is a question based entirely on opinion. The two, however, are historically intertwined. A person cannot have a sincere and effective devotion to their family without a genuine feeling of personal fulfillment.
Like most subjective questions with regard to morality, this question is largely a question of virtue versus consequence. Speaking purely from a moralistic point of view, it is not far-fetched to say that Troy's duty should be completely to his family. After all, it is historically a father and husband's responsibility to take care of his family and nurture them into becoming the best versions of themselves that they can be. However, there is a somewhat obvious contradiction that is easy to see when observing a fictional character. The only reason that Troy is so neglectful of his family is that he feels so personally unfulfilled. The reason he spends so much time engaging in impulsive behavior that pushes his family away is the existential dread that comes with the feelings of having lived a wasted life.
In this regard, it could be said that Troy might have been a more effective patriarch if he had pursued more self-interested objectives, even if it had cost him some time with his family. The overwhelming mistake Troy makes is attempting to be part of a family without himself being a complete human being. His overwhelmingly prominent traits are jealousy and spite, to the point that he cannot even be proud of his son's achievements in athletics. Troy has reached a state of self-loathing wherein he cannot succeed at being a family man or pursuing self-interested goals.