There was plenty of hardship to go around in the former Confederacy at the end of the Civil War, though certainly some populations were hit harder than others.
There were four million former slaves who suddenly found themselves politically free, but socially and economically in almost the exact same situation they faced in slavery. Sharecropping replaced slavery as the main labor system, the Black Codes largely replaced Slave Codes in limiting the rights of freedmen, even requiring in some counties that they be in the regular service of some white person. The Klan emerged as a twisted sort of social police force, and voting rights guaranteed in the 15th Amendment were routinely denied.
Poor white southerners had died by the tens of thousands during the war, with even more wounded. When they returned home, they found a country in poverty, physical destruction of the land, Union occupation and few prospects for the future. They would grind away in poverty, mostly as tenant farmers, for generations.
Plantation owners had perhaps the best prospects, though they too faced economic hardship and Union occupation in the years immediately following the Civil War. Their labor force, slaves, was gone. In some states their crops, homes and cities had been burned and destroyed as had their wealth. Their best remaining hope was that they still owned their land. Within a short time, they had maneuvered former slaves into position as sharecroppers and economic dependents, and reclaimed local government control to rebuild their former system under another name.