The Indians during the time of Indian Removal were not a threat because there were so few of them and they were not generally living a lifestyle that would have threatened whites in the Southeast.
Indian Removal took about 46,000 Indians out of the Southeast. This is a lot, but when you compare it to the non-Indian population of the time, it is not very many. In Georgia alone, the non-Indian population in 1830 was about 516,000. Thus, it is clear that the Indians were not going to overrun the white (and slave) society.
In addition, the Indians who were moved during Indian Removal were not nomads who would wander across Americans' farms or warriors who would raid and kill them. Instead, these were typically people who had pretty much settled down into a "Western civilized" way of life with farms and towns and even black slaves.