Adam Smith was writing against the economic theory usually called "mercantilism" which identified a nation's wealth exclusively with ownership of "bullion" (gold and silver reserves). Mercantilists thus were especially concerned with manipulating balance of trade so that their nation would possess more bullion than other nations and thus be able to pay and field larger armies.
Smith, on the other hand, advanced a concept that is similar to what we now call "Gross Domestic Product", arguing that a nation's wealth should not be measured in bullion alone but in its total annual per capita flows of goods and services.
Since Adam Smith explicitly states that it is the productivity of labor that accounts for the amount of per capita production of goods and services, for him, the wealth of nations is rooted in labor. Increasing the skill level and education of laborers thus increases a nation's wealth.
The factor of production that Adam Smith himself identified as the most important was labor. However, I think that there is a good chance that he would also have identified entrepreneurship as most important if he had known of that concept.
To Smith, the most important source of a nation’s wealth was the ability to take resources and make them into finished goods in an efficient way. In a lot of ways, this is what labor does. If you have a good labor force, it can take resources and work efficiently to make them into finished products. It is much more important to have labor than to have other resources since things like land and capital can be easily bought from other countries.
However, entrepreneurship is extremely important in making resources into finished goods. It is a very specialized form of labor. It is entrepreneurs who take risks and figure out how to make new processes for changing resources into goods. Thus, it is a person like Henry Ford, who figured out how to use assembly lines to make cars efficiently, who is really important to the wealth of a nation.
Thus, Smith would have said that labor is most important and I believe that he would have said that entrepreneurship, as a special form of labor, is particularly important.