I would argue that the cost to the nation would be that the president and executive branch would be less able to do their jobs well. The executive would lose power to Congress and the whole system of separation of powers and checks and balances would be weakened.
Executive privilege is important because it allows the executive to basically tell Congress to mind their own business. It allows the executive to truly be independent when it comes to formulating proposals (as with VP Cheney's talks with the oil company people) or to administering the executive branch. This independence is important because it allows the excutive to act as a true check on Congress; to act with its own truly separate powers.