Jimmy Carter's presidency is seen as a failure by most historians. This is partly because of the Iran hostage crisis, but there is more to it than that.
Carter's presidency coincided with a terrible time in the US economy. This was the era of "stagflation" with high inflation and high unemployment ruining the economy. This was not helped by an oil crisis brought on by OPEC when it cut production and prices doubled. Carter's handling of this is often derided because he gave a speech in which he wore a sweater and told Americans they should turn down their thermostats.
This was seen as a weakness by many Americans who felt that they should not give in and change their lifestyles. Carter's reputation for weakness was also brought on by his attitude towards foreign policy. He believed in an idealistic foreign policy, exemplified by things like giving the Panama Canal back to Panama.