To answer this, you have to think about when Carter was elected. This was in 1976 and it was the first presidential election after the Watergate scandal and the resignation of President Nixon. People wanted someone who did not seem like a typical politician. They wanted someone who seemed honest and down-to-earth. These were the qualities that helped Carter win.
However, those are not necessarily the qualities people want in a president. They want a president who will inspire them and make them feel everything is going to be fine, not someone who tells them hard truths. Therefore, when Carter put on a sweater and made a speech telling people to turn down their thermostats to save energy, he seemed like a defeatist who lacked confidence. What had been refreshing honesty in a candidate seemed like weakness in a leader.
In addition, Carter campaigned as an outsider. This looked good to people who wanted changes to be made in Washington. The problem was that Carter had to, as president, deal with the insiders in Congress. He did not know how to do this at all and he ended up making Congress angry at him instead of getting them to go along with him.
In these ways, Carter's charm as an honest outsider worked against him when it came time to govern.