Consumers in rich countries wearing cheap clothes does not imply exploitation of those in poor countries producing them. A manufacturer could provide standard working conditions and export well made yet low cost clothes, but they may be relatively more expensive to a manufacturer who produces them in substandard working conditions and exploits the workers.
If rich consumers are willing to value the laborer's working conditions, be assured that such are up to standard, and to pay a bit more for the clothes, then the problem is solved. Having voted with money, that company will make a profit and flourish; the one that makes no sale because of worker conditions faces extinction. However, if consumers simply value cheap clothes, that's what they'll get, at the cheapest cost, without regard to working conditions.
Those who want to promote good working conditions must buy the slightly more expensive clothes; they must put their money where their mouth is.
Will most consumers choose to do so?
...The fault lies not in our stars, but in our selves...