A number of different theories have been advanced about the reasons that African people were primarily those who were enslaved. It should be noted that Native Americans also were enslaved. In addition, Africans in Africa owned slaves, and African Americans and Natives in the American colonies did so as well. Therefore, the association of non-whiteness and enslavement does not actually hold up that well.
In terms of arguments for or against enslaving “white” people, these can be divided into heritage and appearance. In the colonies that were settled by people of northern-European heritage, those of the same heritage but who were poor or criminals were frequently bound into long-term contracts of “indenture.” While they might work off the contract over many years, very few were actually able to do that. Although they were not slaves, in that they were still considered persons rather than property, they were bound into highly unfavorable terms committing their labor for what amounted to their whole lives; their debts could be passed to their children, who would assume similar contracts.
Racism against dark-skinned peoples, many of whom were not Christian, is widely considered to have advanced along with slavery, rather than preceding and causing it. Taking people far from their homelands and locating them among people of very different backgrounds was seen as effective in keeping them fearful and obedient; the assumption was that they would be afraid to run away, would have nowhere to go, and would be easy to apprehend because they looked so different from the free colonists. None of those assumptions proved to be entirely true, as the large numbers of people who ran away from slavery and formed “maroon” communities, as well as the increasing number of slave rebellions, attests.