Andrew Jackson had a history of presenting himself as a defender of the common people and an opponent of the rich and moneyed interests. In this, he reflected a divisiveness that had already been present within American politics, by which farmers and laborers had been generally distrustful of the wealthy financial and mercantile interests in the country. (You can see this same theme present in the Washington administration, with the creation of the two party system and the divide between Jefferson and Hamilton.)
For Jackson, then, the Second Bank of the United States was an object of suspicion, one which primarily existed to serve the interests of the wealthy and financial elites instead of the rest of the country, even while it accrued tremendous political power for itself. As if this was not enough, in 1829, seeking to ensure the bank's rechartering, Nicholas Biddle offered to "assume the last of the dwindling national debt to enable its full discharge before the end of Jackson's term, an object that Biddle knew was dear to the president's heart," in exchange for "an early recharter" (Daniel Feller, "King Andrew and the Bank"—see reference link courtesy of the National Endowment of the Humanities). Already hostile and distrustful toward the bank, Jackson read this offer as an attempted "bribe" (Feller's words), which, to Jackson, further confirmed his suspicions.
The 1832 election was largely influenced and shaped by a public discourse surrounding the National Bank, with Andrew Jackson being opposed by Henry Clay. After his re-election, Jackson would continue his opposition to the bank, with its federal charter lapsing in 1836.