I would definitely say you can argue this both ways.
For the good, I think you could say:
- The biggest deal is that he opposed the South during the Nullification Crisis. He did not give in to the idea that the states could nullify laws.
- He gave people a feeling that the president was a common person who cared about them (inauguration party, for example).
- Did not push to annex Texas -- helped avoid conflict over slavery.
- The fact that he ignored the Supreme Court and went ahead with Indian Removal.
- The fact that he made it illegal for abolitionists to send stuff through the mails
- The fact that he put government money in the "pet banks."