With such a decision, I tend to think that there are many elements that a President has to take into account in order to select the people who will be Cabinet members. One of the most important would have to be what values and ideas the specific members bring to the decision making process. The Constitution does not mandate that the President select a Cabinet. It has been understood as part of the job of President that Cabinet members' advice on critical decisions is sought. This would mean that selecting an individual with expertise in the particular post's area would be important. For example, one would not want to pick a civilian who has had no experience with the military for Secretary of Defense.
Along these lines, I think that it would be important for the President to be certain that similar values and shared expectations between s/he and the Cabinet members are present. The individuals chosen to advise the President should have similar ideas and shared understandings about the principles of the administration so that policies can be sought to enhance this base. It becomes problematic if the President and his Cabinet members do not experience that "common ground" element of decision making. The President might value loyalty here, but must do so within the context of ensuring that the best advice in a particular realm can be given.
Finally, I would suggest that a major component that needs to be addressed would be whether a Cabinet member can be confirmed. It is a challenge for a President to find a prospective member of Cabinet who fits all of the needed qualities, but who is not going to be confirmed by the Senate. The President should make sure that who they choose to be part of their Cabinet can be confirmed by members of the Senate in order to avoid embarrassment and in order for a practical reality to be met. This is not saying that the President's decisions are held hostage by Senate confirmation, but saying this needs to be referenced as part of the Cabinet selection process.