There were two different disagreements over this bank.
First, there was the issue of constitutionality. Many Democratic-Republicans felt that the creation of the bank was unconstitutional. They pointed out that the Constitution did not explicitly give Congress the power to create a national bank. They argued for a strict construction of the Constitution while the Federalists wanted a looser construction that would find the bank constitutional.
Second, there was a conflict that came to a head much later over whether the bank was a tool of the elites. Some people, including economic elites, felt the bank was necessary in order to create a stable climate for business in the country. They felt the bank was needed, among other things, in order to keep a strong and sound currency. Meanwhile, other people like Andrew Jackson felt the bank was an elite tool used for helping the elites at the expense of the masses. Therefore, they wanted to kill the bank.