At the start of this essay, Thoreau argues that the war with Mexico proves that the government of the US can be perverted by a relatively few people. To me, he is saying that the war is made possible by the fact that most of the people of Massachusetts (and presumably other states as well) just do not care enough. I believe that it was Edmund Burke who said that all that is required for evil to happen is for good men to do nothing. This is, I think, what Thoreau is saying.
Thoreau says that most people in Massachusetts, at least, do not like slavery. But they do not really care enough to do anything about it. Instead, they just go about their daily lives and don't pay that much attention to slavery. As Thoreau says:
There are thousands who are in opinion opposed to slavery and to the war, who yet in effect do nothing to put an end to them; who, esteeming themselves children of Washington and Franklin, sit down with their hands in their pockets, and say that they know not what to do, and do nothing...
To me, this is why Thoreau thinks the few can win out. The majority dont' care enough to stop the few and so the few can pervert the government.