Conservatives most certainly did not agree with the Philadelphia Plan, though when it was issued it was not such a contentious issue as it would later become. The Philadelphia Plan was essentially the first serious affirmative action plan put forward by the government.
Conservatives are very much opposed to affirmative action programs. They feel first of all that such programs amount to "reverse discrimination" and are therefore morally and legally indefensible. They also tend to believe that market forces can cure discrimination. They argue that firms that discriminate against minorities will do worse than firms that do not discriminate because the discriminators pick less-qualified whites rather than more qualified minorities.
Because conservatives do not believe in government interference in the economy and do not believe in race-conscious programs to combat the effects of past discrimination, they were and are opposed to affirmative action programs like the Philadelphia Plan.