There are any number of reasons why Andrew Jackson inspired admiration and derision, much of which continues, and none of which are mentioned in the answer above.
Among those things that earned derision for Jackson were his war against and ultimate destruction of the Second Bank of the United States. Aside from vetoeing the Bill to recharter the bank, he also withdrew government funds from the bank and deposited them in a number of state "pet banks." His actions were constitutionally questionable, and two of his Treasury Secretaries resigned rather than enforce his order, reminiscent of Nixon's "Saturday Night Massacre."
Jackson's policy dealing with the Cherokee Indians also earned him a great deal of enmity. He defied John Marshall's decisions in Cherokee Nation vs. Georgia and Worcester vs. Georgiawhich ostensibly protected the Cherokee's claim to their lands. Even though the Cherokee made numerous attempts to assimilate into American culture, Jackson would have none of it, and had them transported forcibly to Oklahoma in the "Trail of Tears."
Thirdly, Jackson made frequent use of the Presidential veto which made many enemies for him. He was portrayed in cartoons as "King Andrew the First, of Veto Memory." Opposition to him was so great that a new political party, the Whig party, was formed in opposition to his policies. The name derived from the English party which had opposed the policies of the King.
Jackson receives high marks (and at the time some derision) because of his determination save the Union. He issued his Nullification Proclamation in response to John C. Calhoun's South Carolina Exposition and Protest and let it be known that he would preserve the Union by force if necessary. In one of his most famous pronouncements, at a toast he offered these words: "Our federal union; it must be preserved."
Although Andrew Jackson was not always correct, he was a man of conviction, and as such often created strong feelings. He might best be described in the words of a former Chief Justice of the South Carolina Supreme Court" "I may be in error, but I am never in doubt."
Andrew Jackson was a polarizing figure for two main reasons.
First, he was a very strong personality. In general, it is those leaders who have very strong personalities and who are very aggressive in pushing their agendas who come to polarize the people. People either love them or hate them because there is no question as to where they stand.
Second, Jackson's presidency represented a change in American politics. Jackson represented the change from a time when elites dominated American politics to a more democratic system where the "common man" had his voice heard. Because he represented this important change, which many people would have disliked, Jackson was polarizing.
Andrew Jackson, then, was polarizing because he was a very assertive person with a strong personality who also represented a major change in the politics of the country.