Reconstruction, the period in which the Confederate states were brought back into the Union, lasted until 1877. During that time, the former slaves gained, then lost, many fundamental rights. They were largely losers in the new reality created in the late 1800s. While they were granted the right to citizenship and the right to vote (for men), these rights were slowly taken away in the South. Laws such as vagrancy laws restricted the movements of former slaves, and they, for the most part, did not become land owners but instead worked as sharecroppers. Their right to vote was largely curtailed.
Poor whites were also largely the losers. In the South, they continued to be divided by race so that they supported the white aristocratic class. They worked as tenant farmers or held small landholdings in the South and increasingly worked at factory jobs in the North. If they had joined the former slaves, how do you think this might have changed the situation? Would former slaves have been treated more equally? The winners in this time period were the wealthy white industrial class and white large land holders in the South, who gained more power and wealth.