Civil War prison camps (like Andersonville in Georgia) were generally very terrible places. There was very poor sanitation, and rations were generally insufficient. Huge numbers of men who were taken prisoner died unnecessarily in these camps.
For example, Camp Sumter, a Southern prison camp had about 45,000 prisoners. Of these at least 13,000 died. At Andersonville, 29% of all prisoners died. This was partly because the South did not have enough resources to properly care for the prisoners and partly because of purposeful neglect. The North did not do a great job taking care of its prisoners either. For example, a prison camp in Elmire, NY had 2,963 die of the 12,123 men who were imprisoned there.
I suggest you take a look at the book Don't Know Much About the Civil War, by Kenneth C. Davis. This is where I have found the statistics I have given here (pp. 348-54 in my copy).