The primary reason was television. The visual (and visceral) impact of the Vietnam War was the most extreme in American history because the television brought it into everyone's living room every night.  When Walter Cronkite, the "most trusted face" in the US, said on the evning news that the war was lost people believed him.

The results of the 1968 Tet Offensive did not bear this out.  It was an all-out, committed effort by the Viet Cong and People's Army to end the war at once, and it failed miserably.  The VC were almost completely destroyed by their losses; VC afterwards were PAVN (North Vietnamese) regular soldiers in disguise.  The South Vietnamese Army, cut off from their usual American support, actually fought well, showing themselves equal to the northerners.  In Hue, described in Western media as "the ancient imperial capital" (a city built only in the 18th century), the damage was described by an American officer as, "We had to destroy the city in order to save it."  In reality, there were only small areas of the city destroyed, minor damage compared to any city in Europe in the Second World War.

In Hue, also, the Viet Cong lost their support among the middle classes and intelligentsia.  The very people who welcomed them (intellectuals, college professors, teachers, the educated and liberal part of Vietnamese society) found themselves on pre-prepared death lists.  Most were executed by the VC during their occupation, as well as many young people of near military age, government workers and soldiers on leave from the SVN forces and their families.  They were of course on leave because both sides had previously agreed to a truce during the Lunar New Year, during which the offensive was launched ("Tet" being that holiday).

One thing that swayed American opinion against the war was the famous film clip (and photos) of the Hue police chief shooting a VC prisoner in the head, while the prisoner's hands were bound.  What the news media failed to emphasize was that the police chief had just been informed that his wife and children had been found murdered by the VC.  Unjustifiable as the execution of a POW was, it becomes understandable.

General Westmoreland was, however, thoroughly fooled about Khe Sanh.  While expending vast amounts of ordinance on the areas around the Marine fire base, believing that several divisions of the PAVN were massing for a "final offensive" there in emulation of the 1954 victory over the French at Dien Bien Phu, most of the manpower of those divisions were actually in Hue.  Vo Nguyen Giap, commanding general of the PAVN, used American fears of "a second Dien Bien Phu" to tie down the Marines in the mountainous border area, leaving his troops free to infiltrate further south through the same cavern systems and mountain passes their ancestors had used in the thousand-year war against China and against the French.  It also ate up enormous quantities of airpower, bombs, artillery, etc. on nearly nonexistent targets.

The Tet Offensive of '68 looked a lot worse for the Southern forces and America (and allied) troops than it actually was, but appearances are what matter on television. And, in the end, it didn't matter anyway.  As long as the government of the South was so corrupt and riddled with traitors there was no way to "win" that war.  The tide of Vietnamese history was with the northern forces, and had been since before the Americans arrived.

The Tet Offensive caused a world of doubt to the American public because of the large scale of the attack.  The coordinated advance on different capitals in the region against United States' strongholds caused a rather large level of alarm in the States.  Given the fact that there was a genuine feeling that the war was finally turning in the Americans' favor, the Tet Offensive was a rather shocking turn of events for Americans.  Those political leaders that were slightly ambivalent about the direction of the war were rather solid in stating their dissatisfaction with the war after the Tet Offensive.  The graphic level of images being displayed out of the Offensive as well as the fact that the Americans looked as if they were caught "flat footed" helped to secure more doubt in Americans' mind about the winnability of the war.

General William Westmoreland had recently told Americans on camera that "The light was at the end of the tunnel", meaning that soon we could expect to win and bring the boys home.  He had based this assertion on the fact that Vietcong activity had dropped off nearly everywhere, and he and other military leaders concluded that the counterinsurgency was working, and the war could soon be won.

The Long Range Reconnaissance Patrols (LRRPs) were telling him differently, that there was a river of materials and men moving south.  He missed or ignored that intelligence.  So when the VC attacked every single American military base, the North Vietnamese Army overran the city of Hue, and VC commandos stormed the US embassy in Saigon, it made Westmoreland look very foolish, like he didn't know what was going on.

The Tet Offensive was a major victory for the US on the battlefield.  The VC came out of the jungle and fought on our terms, in the open, and lost badly.  Back home in the US, though, it was a psychological defeat, and a majority of Americans opposed the war for the first time.

The Tet Offensive made the Americans doubt that they could win the war because they had been told at that point that they were winning the war and that the enemy was on the point of giving up.

So they had been believing the Army and the government that said that the enemy was losing.  But then in the Tet Offensive it became clear that the enemy still had a significant ability to fight.

This made Americans doubt their leaders and doubt that they could win the war.