When the outcasts are first kicked out of Poker Flat, Bret Harte notes, "The road to Sandy Bar...lay over a steep mountain range.  It was distant a day's severe travel.  In that advanced season, the party soon passed out of the moist, temperate regions of the foot-hills into the dry, cold, bracing air of the Sierras.  The trail was narrow and difficult."  

First, the fact that the group has to travel over a steep mountain range is enough of a hurdle for anyone, and this group of misfits is not exactly happy about having to leave, so they tend to drag their feet.  Harte also notes how severe the travel will be, and with two women in the group, it is not expected that they will be able to handle the severity of the traveling conditions.  As well, Harte notes that it is "advanced season," meaning that it is likely closer to winter, thus the mountain will be snowy and even harsher to cross than they might in the summer.  Everything about the description of the path from Poker Flat to Sandy Bar seems difficult and arduous.  Mr. Oakhurst knows that if the group cannot get to Sandy Bar in a timely manner, they are ill-equipped to survive in the wilderness.  However, being a gentleman that he is, he also refuses to leave the women in the group to handle the situation on their own, thus he stays with the group when the Duchess halts around noon, declaring that she cannot go any further.