There are two reasons that are generally given for this. First, there is the idea that the successes of FDR's first term had lessened the sense of urgency felt by Congress. Second, there was the court-packing plan that annoyed Congress.
During FDR's first term, the Depression was at its height and Congress was anxious to do anything FDR wanted to try to improve economic conditions. By his second term, things were better (though not by any means completely good) and Congress did not feel the need to do whatever FDR wanted.
Secondly, FDR had annoyed many in Congress with his effort to "pack" the Supreme Court. This made it seem as if FDR was trying to take more power than was rightfully his. This caused something of a backlash in Congress.