In the ante-bellum South, slavery was a legal condition. It was determined by one's status at birth. Unless one was imported, one was born into the status of slavery; and the legal importation of slaves ended in 1808. Gradations of color were common, as many slaves had at least one white parent. For that reason color or wealth had no bearing on one's status. Because slaves were property, the owner normally had a legal deed of title, except for those slaves who were born to slaves he already owned, in which case he had the deed of title to the parent which normally stated that the conveyance ncluded the offspring of the person titled. Legal titles to slaves were typically recorded in public records. Some few slaves were amazingly light complected, but this had no bearing on their legal status or their social standing. Wealth was not normally an issue as slaves were not wealthy. They were normally allowed to keep garden plots and raise some chickens and livestock and keep the money from the sale of these items for their own use. Some managed to buy their freedom or the freedom of their families in this fashion; but the accumulation of wealth was quite anamolous.
The great exception was in the community of freedmen. While slaves were grouped together with no social distinction, freedmen tended to be rather snobbish based on gradation of color. Those who were lighter complected considered themselves the social superiors of those who were of darker color.
Your question raises an interesting point. In other areas at other times, skin color and wealth were determinants; but in the U.S., it was a matter of legal status, nothing more or less.
I think that the issue of slavery was defined by one's ethnicity and the color of one's skin. People of color, specifically those of African descent, were viewed as slaves. It is difficult to ascertain how other "non- white" groups would have been viewed by the Southern landscape of society. Certainly, I think that a case can be made that all people of color would have been subject to discrimination and prejudice. The issue of slavery was something largely reserved for Africans and people from Africa. Those who were slaves were brought from Africa in exchange for goods and services and forced for labor, working and tending to plantations, or domestic help. Slaves were largely deprived of being taught communication skills, as few understood English, and were kept in bondage in order to substantiate White control over them and in order to make their lives miserable in fulfilling social control.
The sub-question of wealth is an interesting one. I think that there are many instances in history where race intermingles with other issues such as class in the narrative of oppression. In this particular instance, I think that race/ ethnicity trumped over all. I don't think that slaves could have "bought" their way out of enslavement. The reality was that the social structure or hierarchy of the South ended with slaves and people of color. They were the bottom, the basis upon which White society of the South were able to assert their own superiority. If this base was able to buy themselves out, then it endangers the entire Southern structure of society. To this end, slavery was defined by ethnicity and race, which became the defining characteristic of Southern society.