The primary "con" to the Bank of the U.S. was Jefferson and Madison's contention that there was no Constitutional mandate to create a bank of the United States. They believed, strongly, that the creation of Banks was property the province of the states and any state bank could properly handle the financial affairs of the country. They cited the 10th Amendment to the Constitution as support:
The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people.
They had no problem with the concept of the bank itself, as did Andrew Jackson much later; they rather believed that creation of the bank would create a dangerous precedent for liberal interpretation of the Constitution. Once that Pandora's box was opened, they believed, there would be no limit to the powers which the Federal Government could claim.
So the true con of the Bank had nothing to do with its operation; it was rather (in Jefferson and Madison's thinking) that by creating it, one would be opening the door to "the danger of liberal construction."
The arguments against the proposed design for the national bank system advanced by Alexander Hamilton have to be viewed from the perspective of the other events and developments taking place at that time. The Constitution was newly ratified and the states were trying to determine how to work together in a unified, national sense to create a new nation.
As Secretary of the Treasury, Hamilton's particular concern was creating a sound financial structure for the new country. He strongly advocated for the government to support business and trade, and used income from taxes levied against whiskey distilled by farmers to help finance that support. His prejudice in favor of commerce and centralized control of government and banking over more democratic and localized policies would be considered the major argument against his recommendations.