The main reason that the Federalists started to fall from power at the end of the century was that George Washington was no longer their leader. Washington's personal charisma and prestige had kept the Federalists together, but now they started to split up.
At first, George Washington was so respected (and the idea of parties was so unpopular) that everyone just stayed in one "party"--the Federalists. This lasted long enough to allow John Adams to be elected as Washington's successor. But then things started to fall apart.
The Federalist faction lost out to the Democratic-Republican faction (as they split apart) because the Federalists were too much of a party of the elites.
So the Federalists lost out because their "party" split after Washington left politics and the Federalist faction was less attractive to people because of its elitist tendencies.