The first answer is absolutely correct. The Jacksonian Era did ignore the rights of women and non-whites.
However, I would argue that it is not ironic to call this the age of the common man.
During the Jacksonan Era, the "common man" really was represented by American government. There was universal white male suffrage and government policies generally followed what common people wanted.
What I'm trying to say is that the first answer is correct about what was ignored. But we must also realize that common men of this era wanted those groups to be ignored.
So, the name "Age of the Common Man" is only ironic if you go by our current understandings of racial and sexual equality. If you compare the era to what had come right before it (or to other countries of the time) it really was the Age of the Common Man.
Indeed, Jacksonian Democracy did increase democratic sentiments with a "common man" touch. It is a bit misleading because some of the elements during this time period did not enhance democratic sensibilities. This can be seen at the start of the election process, where nominating conventions were held to nominate candidates. This was nothing more than a meeting of Jackson supporters who ended up supporting Jackson. This is not an enhancement of democracy, that stipulates the need for discourse and divergent voices.
This theme of "majority only voice" was continued throughout Jackson's presidency. The presence and expansion of the spoils system, where friends and supporters of Jackson, did not increase democracy, as much as it increased patronage and favoritism which are principles against democratic governing. Certainly, the spoils system had been around before Jackson, but Jackson was so brazen about its growth and enhancement, almost to the point where he actually argued that it increased democracy.
Another element within Jacksonian Democracy which actually inverted democracy was that Jackson did not seek to enhance dialogue and diversity in thought. Through the spoils system and the brash manner with which he carried himself, Jackson did not seek to bring dialogue or discourse by the encouragement of dissenting voices. Rather, he and his followers created a feeling of an almost "regime- like" sensation as government positions were stacked with Jackson loyalists.
Additionally, Jackson never sought to integrate the other branches of government within his presidency tenure. This is proven in Jackson's disagreement with the Supreme Court case of Worcester v. Georgia, that held that Native Americans had a right to live on their land specified in treaties with the federal government. Jackson, seeking to move the Native Americans out, did not enforce the decision with the necessary zeal of a President in following the orders of the Supreme Court.
Finally, Jackson's forcible movement of Native Americans, the relegation to reservations, and the Trail of Tears are examples where it is hard to conceive of Jackson acting in democratic interests or speaking out for "the common man." By itself, Jackson's treatment of Native Americans can be seen as an example of acting against the interests of democracy and democratic sensibilities.