To understand why this is so, let us first look at what Jefferson’s ideals were. The relevant ideal that Jefferson held was an image of the United States as a country of small, independent farmers. Jefferson believed that this was the only way to have a true democracy. He believed that small farmers were not dependent on anyone else (since they grew and made the vast majority of the things they needed). Therefore, their votes could not be swayed by anyone else. This made them worthy of participating in a democracy. Jefferson felt that anyone who worked for someone else, or who made their living selling things to others, was essentially dependent on those other people and could not be truly equal to them.
The Second Bank of the United States did not contribute to such a society. Instead, it was created mainly to help create a Hamiltonian vision of a society in which there were factories and many merchants along with farmers. This is not what Jefferson wanted. President Jackson also believed that the government should work for the common person, not for the elites. He wanted a society similar to that which Jefferson wanted. When he vetoed the bank, he was (he felt) striking a blow for an egalitarian society of small farmers. This is exactly what Jefferson would have wanted.