Colonel Sherburn's attitude towards these men is one of total contempt, just like he had for Boggs.  He does not think that any of the people who are talking about lynching him have the guts to actually do it.  So he looks down on them because he thinks that they are inferior to him in terms of their will and their bravery.

You can see this in Chapter 22.  He looks at them and he talks to them scornfully, telling them that they are not the sort to every lynch anyone like him.

did that make you think you had grit enough to lay your hands on a man? Why, a man's safe in the hands of ten thousand of your kind -- as long as it's daytime and you're not behind him.