The Constitution was a delicate balance between the needs of the states and the need for more power for the federal government. The Founding Fathers were aware of the need for a stronger federal government and military after Shays' Rebellion of 1786-1787. However, the states and people were wary of a government that was too powerful, as they had just broken away from what they considered to be the dictatorial powers of the English crown.
To forge a compromise, the Founding Fathers gave some powers to the federal government to provide for a more efficient functioning of government, while leaving other powers to the states. For example, only the Congress has the power to regulate interstate commerce and make treaties with foreign powers. Relegating these powers to the federal government makes the relationship of states with other states and the relationship between the federal government and foreign countries smoother and more efficient.
In addition, the Founding Fathers wrote the Constitution with the idea of checks and balances in mind. For example, each branch of government has the ability to check the other branches (the President, for example, can veto a bill from Congress, and Congress can override that veto). These ideas came from Montesquieu's idea of the separation of powers.
Finally, the Founding Fathers had to forge several other kinds of compromises. For example, the Congress was divided into two houses (the Senate and House of Representatives) to allow for a compromise between small and big states (small states receive equal representation in the Senate, but larger states have more representation in the House). Slavery was also a contentious issue, and the Founding Fathers allowed slaves to be counted for 3/5 of a person to satisfy the southern states, who wanted to count slaves for representation but not for taxation. In addition, the Constitution stated that the slave trade could not be stopped until 1808, twenty years after the writing of the document.