I would argue that the geography of slavery made the South feel more and more in danger from the North.
As you say, it was generally believed that slavery could not thrive in the Southwest. This meant that slavery could not spread west because there was no area past Texas where it could be viable.
Because of this, the South would have felt like their political power was going to be lost. As the US expanded, all of the new territory was likely to be free because it was either in the north or no good for slavery (the SW). This lent a sense of desperation to the South and made it more likely that they would try to secede.