What might have happened to slavery in the South if Eli Whitney had not invented the cotton gin?
Eli Whitney did not invent the cotton gin. Such devices had been used as early back as the late 1600's to separate seed from fiber. The critical modification Whitney made was building a device that could separate these quickly, but in the process, destroyed some of the cotton fiber, which yielded a lower quality cotton. Because the demand for any kind of cotton was so high, his modification made "cheap" cotton available, a lesser costing and lesser quality variant of what had been sold. The demand for cheaper American cotton therefore increased, so plantations grew more of it.
Had he not made the "improvement" to the device, slavery still would have existed, but slaves would have been farming tobacco or rice rather than cotton. Although the demand for cheap American cotton would have increased the demand for slave labor, it also increased the demand for Whitney's machine. Ironically, if slavery had been on the increase, the mechanization of agriculture quickly made a slave more expensive than the machine, and plantation owners would have opted for mechanization. Had Whitney made his modification a decade earlier, the issue of slavery would have been moot, as economics would have driven it to extinction.
The cotton gin was only one factor that caused the rapid increase in slavery. The other, more important factor in my opinion was the the addition of vast tracts of new farmland as the number of states and territories increased and moved west. This caused slavery to relocate as well, or at least to move its economic center. It's also important to remember that the South had no economic alternatives to plantation slavery by the early 1800's, so its expansion as an institution was a foregone conclusion.
The main impact of the cotton gin was that it made cotton plantations much more profitable and accelerated this growth in slavery. The institution itself would have been around a long time regardless of Whitney's invention. We may have seen other cash crops such as sugar and rice make an appearance in wet, humid places like Louisiana had the gin not been invented.
Of course, this is only speculation, but it might well have died out, or at least not gotten to be nearly as big as it was.
Slavery in the Southern US was in trouble before Whitney invented the cotton gin. It had been based mainly on crops, like tobacco, that were no longer doing very well in the US, economically. This meant that there was coming to be much less of a need for slaves.
When Whitney invented the cotton gin, it caused a boom in cotton production and therefore a boom in the demand for slaves.
So if he had not invented his machine, slavery might have died out. I doubt it would have done much good for the slaves, though. I imagine they would have been sold to the Caribbean, not freed.