It is true that imperialism and colonization have left behind a linguistic and religious legacy. However, we also have to look at the legacy that colonization left in terms of government and the economy. This legacy has, many social scientists argue, made it harder for countries that had been colonized to have stable governments and strong economies.
Most countries that used to be colonies have weak economies and unstable governments today. (When I talk about colonies in this question, I am referring to countries where an imperial power came in and dominated, but did not bring large numbers of people. Such countries are still largely made up of indigenous people. Thus, the US is not counted as a colonized country while the Philippines and Nigeria are.) This is largely, we can argue, due to colonialism.
When imperial/colonial powers took countries over, they used those countries for their own benefit. This meant that they created economies that were meant to help the colonial power, not the colony itself. This has continued to cause weakness in the previously colonized countries’ economies. They were set up merely to feed raw materials into the colonizer’s economy. This means that their economies continue to be based on extractive industries which are very low value-added. Social scientists sometimes say that these economies are “neo-colonial” because they still operate mainly for the benefit of the rich world.
Imperial powers also did not set their colonies up to be able to govern themselves. They did not give the natives good educations. They did not train them to be part of government. They did not allow them to govern themselves, thus creating the habits needed for democracy to thrive later on. This is a major reason why such countries have been largely unstable. They do not have the experiences needed to make it easy for them to have stable, democratic governments.
In these ways, the legacy of colonialism goes well beyond religion and language. Instead, it continues to retard development in countries that were colonized.