Andrew Jackson hated the Bank of the United States because he felt that it was a tool of the elites that would be used to harm the common people. He also felt the bank was corrupt and that it was the creature of foreign investors. Therefore, when Henry Clay pushed a recharter of the Bank through Congress, Jackson vetoed it. This was in 1832.
After Jackson was reelected later that year, he decided to really go after the Bank. It was at this point that he started to remove federal money from the Bank. He took the money out of the Bank of the United States and put it into several "pet banks" chartered by various states. This harmed the Bank severely and it ended up dying when its charter expired in 1836.