This is said because Speer was one of the leaders most responsible for using slave labor to run the German economy during the war.

Speer was made Minister of Armaments and Munitions in 1942, later taking on the title of Minister of Armaments and War Production.  His job was to keep the German economy producing as much as possible so that it could fight the war.  During the war, most German men were, of course, fighting.  Hitler did not want to use women to work in the factories.  This caused a problem which Speer solved by using slave labor.  He used prisoners of war and he used people who had been taken from the occupied countries of Europe and brought to Germany to work.  

For this reason, he was one of the "true architects" of slave labor during the war.