The answer to this depends in large part on what your own skills and aptitudes are. I have taught many students in economics classes (high school and first year college) over the years, and students who are good at math have a hard time with different things than those who are better in English or history.
For those who are good at history, the hardest aspect economics is undoubtedly the math and the graphing. It is not that the math is hard. At that level, you are not doing anything as hard as algebra. The problem is simply that many of these students are frightened by math and by graphs. Since economics at this level involves a good deal of both math and graphing, these students often struggle.
For those who are good at math, the harder part is understanding how to apply the math to real world problems. In other words, what they often struggle with is all the “story problems” that are involved in economics. For such people, the math is really simple. The problem is that they have a hard time figuring out which rules or formulas apply to specific scenarios.