Looking at and interpreting a graph from Macroeconomics by McEachern, I would argue that most of the federal debt in 1980 came as a result of defense spending.
Defense spending made up around 50% of all federal outlays in 1960. This declined, especially after the end of the Vietnam War, but it was still around 25% of all outlays in 1980. For the whole period 1960 to 1980, defense spending is the single biggest category of outlay. Moreover, defense spending was surely a bigger portion of federal spending during WWII and the Korean War--especially since there were fewer social programs during those days.
So, looking at that graph, I think that the best answer is that defense spending was the cause of most of the debt before 1980.