This is an age old question that historians have debated hotly for many years. The traditional justification for black slavery was based on the story of Noah who, after leaving the ark, built a vineyard, got drunk and passed out. While he was unconscious, his son Ham apparently did something improper to him for which Noah cursed Ham's descendants. Many old scholars insisted that Blacks were the descendants of the biblical Ham. In fact, in Uncle Tom's Cabin, a minister sternly admonishes little Eva, "cursed by Canaan."
The Pope at one point authorized enslavement of Blacks because they were not Christian. A similar practice was carried out in the Muslim world: the Koran forbade enslaving a believer, so Muslims took the position that enslavement of non-believers was perfectly justified.
I have published online a lengthy research paper on this topic. I have referenced it in the second link below. Please consult the enotes link first; afterwards, if you read my research, it may enlighten you further than I can in a single post.
I think the fact that slavery in the United States consisted of mainly African Americans is that the Triangular Trade Routes brought the slaves to the United States from Africa.
I think the recent(ish) film Apocalypto directed by Mel Gibson was really interesting in the way it showed that slavery and oppression were not colonial inventions, but were natural outcomes of one nation showing its strength over another. Postcolonialism offers one convincing approach to explaining the link between race and skin colour by its theory of "othering," which is the term given to the way that we define ourselves in opposition to another group. Therefore, "we" are civilised, whereas "they" are ignorant and stupid. We are powerful and mighty, whereas they are weak. Such thinking quickly leads to a sense of superiority, and eventually, slavery.
I'm glad the above post mentioned ancient slavery. Ultimately slavery is based on power and frankly on the idea that the "masters" are in some way better than the slaves. In ancient Rome it was the conquered peoples who became slaves. "Modern" enslavement of blacks was also based on the idea that the Europeans were better than the peoples of Africa and later the West Indies. Slavery is based on a balance of power that needs a strong belief behind it to maintain that power. You have to have a reason to enslave one person rather than another. Skin color is an obvious difference. It is easy to say "look, they are not like us, they must not be as good as us" because the differences are very obvious.
Slavery has always been forced upon peoples who have been conquered...it's as old as humanity. So, therefore, I don't think it is entirely race-based--sometimes based more on religion. Spartans, Trojans, other countries enslaved white people simply because they were the conquered. They also frequently captured people of other races and conquered peoples for the purpose of fighting in the colosseums for the entertainment of the masses.
Slavery has mostly been based on power...not so much race...although more modern cases of slavery have been more black people enslaved than any other large group. Keep in mind, that many of these black people who were captured in Africa were captured by other black tribes and sold to the Portuguese and others who offered money for this purpose. The strong "captured" the weak were enslaved and seen as "less than".
There are many different views on this subject.
One view holds that slavery became racialized simply because having slavery limited to blacks was convenient. Blacks were seen as more convenient slaves for a number of reasons. They were conspicuous, for one thing. If people could assume that almost all blacks (and only blacks) were slaves, it would be much easier to keep control over the slave populations. In addition, blacks imported from Africa would be much less likely to demand rights than white indentured servants would be.
Another view is more of a Marxist one. This one can be seen, for example, in the writings of Ronald Takaki. Takaki argues that it was in the interests of the upper class to keep the lower/slave classes divided. If slavery and/or wage labor were a multi-racial thing, that class might develop solidarity and throw off their oppressors. By defining slavery in racial terms, the upper class was able to use race to divide the lower classes against one another and rule them. To this way of thinking, slavery became racialized because it helped the upper classes maintain their dominance over the lower classes.