Compare the role of the Cabinet with that of the Executive Office of the President.
Typically, in modern American presidential administrations, the Executive Office is much more important to the President than is his (or maybe someday her) cabinet. This is because the members of the Executive Office are not divided in their loyalties.
One of the things that a president most prizes is loyalty. They want to be sure that their assistants are really out to help them and no one else. They can be relatively sure that members of the Executive Office are loyal because they have no boss other than the President.
By contrast, members of the Cabinet often have divided loyalties. They are confirmed by the Senate, for one thing. They also have to deal with the agencies they head and with the constituences for those agencies. This means that they may come to care more about the agenda of their agency or its constituents than they do about the President's agenda.