Modern imperialism in the form of international corporate/banking control is equally damaging everywhere. In historic terms, it would be hard to make a case for this question's assumption; Africa is without doubt the most wrecked part of the world today, and that is in large part due to colonialism. The boundaries of African countries were drawn up by European powers with no regard to ethnic, cultural, religious or linguistic reference. The very end of colonialism in Africa was done in such haste after WW II that almost no African country had any real ability to run itself or its political, economic, military or social issues. This was not because of some mythical inability of Africans, but because the European rulers had never really integrated Africans in the processes of governing or in the running of the economic life of the nations. The Congo is the most glaring example, in some respects, but no more really than Angola or many others.
Colonialism's main negative impact on South America is that the countries of the continent were mostly under Spain during colonialism, and therefore have several major problems inherent from that. One, they have Spanish-based law, meaning the accused must prove their innocence in criminal cases, infinitely more difficult than defending themselves while the State must prove their guilt. Two, the legacy of military control and rule by an elite minority, usually of lighter-skinned, more ethnically Spanish people. That goes along with the third, the extreme inequality of wealth, with a small minority of rich and powerful families ruling in a business-consortium manner, with a large mass of poor peasants and a small middle class. The fourth is more subtle, the tendency of Spanish-based cultures to attempt to redress grievances by revolutionary activities. This is in part due to the elitist nature of Latin American society and government and resistance to the needs of the people, but in part is a legacy of the Napoleonic Wars. Spain's military was unable to resist Napoleon's forces, but with British help they waged a very successful guerrilla war. The problem is that this left Spain (and Latin America) with a tradition of guerrilla warfare when dealing with any major governmental problem which frustrated a major portion of the population.
There is no doubt that colonialism left S. America with a legacy of serious problems, and that modern "globalization" is wrecking the economies of the continent. But in neither case are those effects really worse on that continent than in Asia, certainly not worse than in Africa.