It should be pointed out that the Election of 1824 was one of two decided by the House of Representatives, as no one had a clear majority in the Electoral College, although Andrew Jackson did hold a plurality of the votes. Jackson's opponents in that election were John Quincy Adams, Henry Clay, and John C. Calhoun. Calhoun withdrew from the race and ran unopposed for Vice President, leaving three men in contention. As noted above, Clay also withdrew and threw his support behind Adams, and was appointed Adams' Secretary of State. It was not the prestige of the office of Secretary that infuriated Jackson; it was the fact that the office of Secretary of State was considered a stepping stone to the Presidency. Several previous presidents (Jefferson, Madison, Monroe and Adams) had all served as Secretaries of State; therefore it appeared to Jackson that Adams and Clay had virtually locked him out of the Presidency for anywhere from eight to sixteen years. It should be noted that although the "corrupt bargain" position played well with the public and was used effectively by Jackson and Martin Van Buren, such a "bargain" was unlikely. John Quincy Adams was a principled man who hated politics and political dealing. It would be completely out of character for him to make such an arrangement. Also, Henry Clay had a visceral hatred of Andrew Jackson and opposed him at every turn, particularly while Jackson was President.
It was not so much Jackson's supporters who were infuriated as it was Jackson himself. He began his campaign for the 1828 election almost immediately after the election of 1824 was elected. Although electors were still chosen by state legislators, Jackson carried his campaign to the people in the first truly modern electoral campaign. Speeches, barbecues, fireworks, etc. kept his name before the public (and through them the electors) such that he defeated Adams handily in the 1828 election.