In Zinn's A People's History Why did Virginians massacre Indians instead of enslaving them?
During the Age of Discovery, Christian Europeans of all nationalities held largely similar attitudes toward Native Americans; they were, at best, misguided and childish, and at worst, Satan-worshipping cannibals. The only problem was reconciling this prejudice with reality; the Native Americans knew the land better than any settler, were often generous and rational in their diplomacy, and Europeans had a disturbing habit of "running off to join them" when times were hard. A variety of competing economic, religious and social pressures resulted in the common solution of extermination, as Zinn discusses throughout Chapters 1 to 4, but most specifically Chapter 1. He succinctly states;
Not able to enslave the Indians, and not able to live with them, the English decided to exterminate them.
Why were they not able to enslave, or live with, the Indians?
One near-insurmountable difficulty in enslaving Native Americans was their canniness, which has often gone unrecognized in stereotyped accounts. While it is true that some Natives initially mistook Europeans for gods or other supernatural beings (Zinn gives the example of Cortes being mistaken for Quetzalcoatl), this was almost always dispelled rather quickly, though not always in time for the Natives to save themselves from the robbery and extortion settlers had a habit of employing. What the real problem was, was the Native's abhorrence for direct engagement; they would not fight the way Europeans fought. Even during the French & Indian and Revolutionary wars, Native Americans were known as excellent scouts, but awful frontline combatants. Far from being cowards, this was simply the way that Natives fought, and a wise response to the superiority of European firearms.
Thus, how could one enslave a population that was aware of your intentions, knew the land better than you did, and had an annoying habit of avoiding direct confrontation? The answer is, you couldn't, at least not on a very large scale. Native enslavement was not unheard of, but Native slaves were often worked to death, which led to the employment of African slaves, as Zinn explains later.
Of course, that begs the question of how you're supposed to exterminate an enemy that you can't find. In the end, smallpox did the job better than any gun could have; most Native American populations suffered enormous (more than 75% of some populations) losses to disease.
That being said, why was living with the Natives impossible? This has more to do with social upkeep; it was demoralizing and inconceivable, to some, that any degree of misfortune would lead a "civilized" European to debase themselves by living with savages. Many times over the Europeans reiterated their "civilized" rights, compared to the "natural" rights of Natives, which were really just euphemisms justifying the settler's humanitarian crimes. How, then, could this perspective be maintained if settlers were learning from, living with, and even marrying or having children with such savages? Zinn also extensively describes the daily lives and moral laws of various Natives near the end of the chapter, setting them in contrast to European values. Any time two cultures directly interact, conflict will inevitably ensue.
Thus, the Natives were exterminated largely because of culture conflict, specifically the less-tolerant European attitudes demanding their subjugation, coupled with the Native's ability to elude direct engagements with European military and political power.