When slavery became a common thing among the Colonies, and then in the established United States, many landowners had slaves imported in quantities that overwhelmed the number of white citizens. It was not uncommon for a plantation in the South to have more slaves than whites, and it was mostly the threat of retribution and the general bias against blacks that kept slaves from revolting. It was hard for many slaves to accept their status, especially when they outnumbered whites, but until Emancipation and the Civil War there was no outlet for slaves to really become free.
Probably the largest concern for slave-owners in the South was the possibility that slaves would revolt against their owners and take revenge for mistreatment. Local government allowed atrocities against blacks without punishment, so it was easy to understand why slaves would feel the need for revenge. However, the allowance of retribution against slaves -- and blacks in general -- was always a threat, and so while there were significant revolts -- such as that of Nat Turner in South Carolina in 1891 -- the institution remained intact.
Another concern was the reproduction and spread of blacks; the public bias against non-white races meant that although black slaves were considered necessary for work, they were not wanted as citizens; births were often aborted because the baby would curtail the mother's work while putting another burden on the slave-owner.