These two works are similar in that they are major works of political thought, but they are not really trying to address the same issue. Rousseau's work is trying to look at the bases for society -- why society comes to exist. Montesquieu, by contrast, is trying for a more applied approach. He is trying to look at how a society (once it is created) should be governed.
Rousseau was concerned with why societies form. He was thinking about the state of nature and how society forms as a response to this state. Because of this, he was laying down only the most basic ideas about the actual governing of a society.
Montesquieu, on the other hand, was trying to look at the political institutions that societies have. He was trying to show which political institutions were best. This is a more applied book -- he is looking at the actual workings of government as opposed to the theories of why governments exist.