The men refuse to go with the dwarves because they are afraid of the dragon.
It is noted that the men are more ready to believe in the dragon than in Thorin. They are concerned about the legends of old, and believe that the dwarves’ mission is doomed.
They packed what they could on the ponies and the rest was made into a store under a tent, but none of the men of the town would stay with them even for the night so near the shadow of the Mountain. (ch 11)
The men had lived in the shadow of the mountain hearing stories about the dragon for years. They had only recently been faced with the possibility of the dwarves overcoming the dragon. They believed the songs were true and the dwarves would be killed, and therefore did not want to go down in flames with them. They supported them and cheered them on, but were not willing to go along.
The men’s unwillingness to accompany the dwarves demonstrates again how this fight is really the dwarves’ alone. They are fighting for their home, because they have no home to go to. They are trying to get back what they lost, so in some ways they have nothing to lose.