The primary "con" to the Bank of the U.S. was Jefferson and Madison's contention that there was no Constitutional mandate to create a bank of the United States. They believed, strongly, that the creation of Banks was property the province of the states and any state bank could properly handle the financial affairs of the country. They cited the 10th Amendment to the Constitution as support:
The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people.
They had no problem with the concept of the bank itself, as did Andrew Jackson much later; they rather believed that creation of the bank would create a dangerous precedent for liberal interpretation of the Constitution. Once that Pandora's box was opened, they believed, there would be no limit to the powers which the Federal Government could claim.
So the true con of the Bank had nothing to do with its operation; it was rather (in Jefferson and Madison's thinking) that by creating it, one would be opening the door to "the danger of liberal construction."