Slavery is said to have restricted the South's ability to diversify for two reasons.
First, it is said that slaves would have made bad industrial workers. Slaves on plantations worked slowly and broke their tools as often as they could as a way of pushing back against their owners. Slaves in factories could have done the same, causing much more destruction as they broke costly machinery.
Second, slavery meant that the South's capital was tied up in human beings. Slaves cost a lot of money. This meant that Southerners had to sink most of their money into buying slaves and therefore lacked the resources to do things like investing in factories.
Slavery was essentially a system of feudalism in which one class (planters) relied on unfree labor. Without free labor, the South did not develop into an industrial or fully capitalist society. For example, there was very little industry in the South by the outbreak of the Civil War, and there was also very little railroad trackage in the South as compared to the North. In addition, the banking system in the South was very rudimentary, and much of the South's funding came from the North. The cotton harvested by slave labor in the South was turned into textiles in the North or in Europe, and there were few textile mills in the South.
The profitability of cotton, particularly after the cotton gin was invented by Eli Whitney in 1793, and the widespread use of slavery meant that the South was wedded to what it called "King Cotton." Some voices in the South, such as J.D.B. DeBow (the publisher of DeBow's Review) urged the South to move away from agrarianism and become more industrialized, but the South was still a cotton-based economy at the time of the Civil War.