In The Wealth of Nations, why does Smith think that pursuing one's personal interest is a more effective way to benefit society than trying to pursue the interest of society as a whole?
The idea that pursuing private interests can lead to public benefits was first advanced some seventy years before Smith's Wealth of Nation in The Fable of The Bees: or, Private Vices, Public Benefits by the Anglo-Dutch philosopher Bernard Mandeville. The key revolutionary economic idea of this work was that spending, including spending on luxuries, provided a greater economic stimulus than saving. Thus, what was typically considered prudent and morally good—the life of restraint and thrift—actually harmed the economy, while spendthrifts helped the economy. The aristocrat who threw lavish parties, owned large stables of horses, and ordered expensive clothing, lavish artwork, home furnishings, and jewelry provided work for far more people than the one who lived simply and saved money.
While Mandeville's work was considered scandalous in his time, Smith expanded and refined some of Mandeville's ideas and, from that kernel, created a system of economic thought that accounted for how markets were self-regulating.
The first major element of Adam Smith's account is that he softened the traditional Calvinist belief of man's "total depravity" to argue that most people were, by nature, benevolent. Thus, even pure self-interest would be tempered by reason and fellow-feeling. He also saw humans as rational agents who would make decisions while considering long-term effects of behavior. Stealing, for example, would not be considered an example of self-interest because of both the negative effects when one is caught and because living in a society where stealing is common is not actually in our self-interest.
As a consequence of his view of human nature and a theology that saw God as having created a rational universe that followed universal laws and principles, Smith saw economies as self-regulating under the conditions of free trade and open markets. This would also increase prosperity. If, for example, someone paid workers badly, the workers would take other jobs that paid more. If a company tried to overcharge for an item, another company would move in and undercut prices. Thus, pursuing self-interest, in the form of trying to maximize one's own wages or increase productivity and profitability, benefited the economy as a whole. Monopolies and government interference both limit this self-regulating and autonomous "invisible hand."
According to Smith, when people all pursue their personal interests, society benefits through the "invisible hand." This is, to Smith, much more effective than efforts to consciously improve society.
Smith believes that people, pursuing their interests, will improve society. As they pursue their interests, they will do things that make money for them. They will do things like starting businesses and pursuing profit. When they start businesses, they will eventually employ people. As they pursue profit, they make money that will be used to buy goods from other people, once again benefitting society.
All of this is better than having conscious efforts to improve society. This is because no one person or small group of people is intelligent enough to know how to improve society. It is better if everyone pursues their interest and allows the "invisible hand" of competition to improve the society.