The war was described in this way because, particularly in comparison to wars the US had previously fought, it was unconventional and hard to understand.
In World War II, for example, there was a clear form to the war and a clear front. The Allies won the war by gradually taking more and more territory away from the Axis. For example, a person could follow the course of the war in the Pacific as the Americans moved closer to Japan, island by island. It was clear when progress was made and it was clear which side was winning. There was a clear objective and the fighting was generally taking place in a fairly defined area.
In Vietnam, by contrast, none of this was true. It was not a war where there was a specific territorial objective. The US could not win the war by taking a given city or a region. Thus, the war was formless because there was no specific objective that could be shown on a map. It was not possible to see progress towards victory. There was no front line because fighting could occur anywhere at any time. Therefore, the Vietnam War was completely different from previous wars and it could be called “formless” and “frontless.”