The major reason for James Madison's ineffectiveness as president is that he was president at a very bad time.
Madison, of course, succeeded Thomas Jefferson. Jefferson was a towering figure in his political party and, while he was leader, various factions within the party kept their conflicts under wraps. After Jefferson was gone, however, the factions started to quarrel openly. This meant that Jefferson did not have a united party behind him.
Madison was also handicapped by the fact that the war was going on between France and England. This war put the US in a very difficult position. In addition, Jefferson's Embargo Act had harmed the US economy and had to be deal with. This handed Madison some serious problems at a time when his party was not united to help him solve the problems.
When you add these problems to the fact that Madison was not personally a great leader, you have the recipe for an ineffective presidency.