The Federalists were the first political party of the new American state, and came into being over the expressed opposition to parties of many founding fathers. Washington accepted two terms as president hoping to avoid party politics entirely, without success. To a large degree the Federalist, Whig and modern Republican Parties have been actually the same party with changes of name.
The Federalists believed not only in a strong central government but in the rule of a small economic and political elite. In many respects they were more oligarchical than democratic or republican in outlook. The party coalesced around Alexander Hamilton, whom they believed would be highly influential because of his long association with George Washington as a staff officer during the Revolution. Their platform of tight economic control and the doctrine of "implied powers" given by the Constitution to the central government ensured the opposition of the majority of the signatories of that document, including Madison, Jefferson, etc. But as time passed the Federalists became a "state's rights" party and the Democratic-Republicans used the argument of implied powers to legitimise the Louisiana Purchase. Opposition to the War of 1812 spelled the end of the Federalists.
The Whig Party, established 1834, took their name from the anti-royalist party in Britain, who have since become the Conservatives. The Whigs in America were largely opponents of Andrew Jackson's policies. Henry Clay and Daniel Webster were the best known members. Harrison and Zachary Taylor were the only Whig candidates elected president, Taylor after his victories in Texas during the Mexican-American War. The party line was essentially the same as the Federalists' had been.
The Republican Party began in the 1850s, from remnants of the Whig and Free-Soil Parties and disgruntled Democrats. The Democratic Party, of course, was Jefferson's old Democratic Republicans, and the "new" Republican Party took their name in an attempt to drape themselves with the legitimacy of the older party. They were also called the G.O.P., or "Grand Old Party" to connect their history with that of Hamilton and Adams, etc. The party grew out of the struggles involving the Missouri-Kansas border war and the Kansas-Nebraska crisis, ostensibly about the issue of slavery. The depredations along the borders were actually carried out by bandits using pro- and anti-abolitionist propaganda as a cloak for their activities, but it did make slavery a burning national issue.
Oddly enough, although Abraham Lincoln was the first Republican president he was also the only third-party candidate to be elected president in the US. He ran for reelection on the "New Republic" ticket, with a Democrat (Andrew Johnson) as his running mate. Lincoln had become disenchanted with the Republican Party while president, much like Eisenhower in the 1950s.