In my opinion, the two most immediate dangers to the outcasts are exposure and lack of food.
Oakhurst, the Duchess, Mother Shipton, and Uncle Billy are thrown out of Poker Flats because they are considered amoral townsfolk. The people of Poker Flats are nice enough to provide the group with horses and mules, but food is not provided. The group decides to go to Sandy Bar, a town that is a long day's ride away. Not having food wouldn't be a problem, even if it took two days. People can survive much longer with no food. However, the absence of food plays an important part in the story once the snow storm hits and prohibits the group from continuing on.
Exposure is the bigger concern. "In that advanced season, the party soon passed out of the moist, temperate regions of the foothills into the dry, cold, bracing air of the Sierras." It is winter or late fall, but the author makes it clear that being out over night in the high Sierras is a dangerous gamble. The group is ill prepared in terms of having food on hand or proper cold weather clothing in case a snow storm hits. The story seems to indicate that the group was escorted out of Poker Flat with nothing but the clothes on their backs. They have no way of surviving in those mountains if something goes wrong. Oakhurst knows this, which is why he is against stopping in the first place.