The most important potential down side to ending subsidies for agriculture would be that some agricultural businesses (whether large businesses or small farmers) might be forced to leave the industry. It is possible that many farmers would be unable to make a consistent profit without these subsidies. If the subsidies were completely done away with, the farmers might go out of business.
The question then, is why that would be a problem. We let other kinds of businesses fail all the time, so why prop up farmers? There are at least two reasons for this. First, there is the idea of food security. If we lost our domestic farming industry, we would be too susceptible to pressure from outside countries. If, for example, there were a war or a conflict with some country that exports a lot of food to us, we might be in trouble because we would not be able to grow our own food. Second, there is the idea that farmers are the backbone of America, particularly in rural areas. If farms went out of business, rural areas would be economically devastated. In addition, something of our heritage would be lost because our country was built (in this view) on family farms.
So, while many people say ending subsidies would be good, others say it would be bad for strategic, economic, and social/cultural reasons.