Slaveholding was never particularly common, even in the South. The vast majority of white people in the South did not own any slaves at all. About three quarters of all white Southerners did not own any slaves and did not belong to a family that owned any slaves.
In 1860, it is estimated that about 385,000 families in the South owned slaves. (This statistic, as well as the one in the previous paragraph, can be found in the link below.) If, then, you define a slaveholder as the head of each of these families, then there were 385,000 slaveholders in the United States in 1860.