If it were not for the fact that slavery was involved, it would not be hard at all to agree with the South's reasoning. Their justification was not unassailable, but it was defensible.
The major justification for secession was that the Constitution was a pact between states, not between the people of the country as a whole. Perhaps the easiest way to support this is to point out that ratification was not done by a vote of all the people. Instead, it was done on a state by state basis. This implies that each state was consenting to join the Union. If a state could consent to join, then presumably it could withdraw that consent as well. After all, government is only valid (the Declaration of Independence tells us) when it is done with the consent of the governed.
Because this argument is so closely associated with slavery, it is hard to be sympathetic to it today. However, it is not a ridiculous argument.