In general, Andrew Johnson drove the moderate Republicans in Congress to cooperate with the radicals when he proposed Reconstruction policies that were too lenient to the South.
For example, Johnson's plan did not force many of the men who had led the South to step down from their positions after the war. He pardoned so many ex-Confederates that even moderate Republicans came to oppose him.
In addition, Johnson did some things that made the moderates question his sense. In one instance, Johnson gave a speech in which he criticized the Congressional leaders of Reconstruction very harshly and even implied that they had thrown out the whole concept that the Union was indestructible. In other words, he was accusing Congress of throwing out the whole idea behind the Civil War. This was a very inflammatory thing to say and it helped to push moderates even farther away from him.