The real problem with siding with Jack at the vote was that they would have to say no to Ralph. This is a big deal for two reasons. First, up until this point, Ralph is their voice of authority. Whether or not you agree with authority figures, it is always extremely difficult to look at them (especially as kids) and tell them they are wrong. Secondly, Ralph has made it clear that his way of leadership will get them saved, and they will stay alive. Jack hasn't really made those claims yet. Any vote against Ralph would be a vote against going home - not something you really want to admit in front of the group.
However, the allure of fun and games with Jack is just too much to avoid for too long. Once they aren't under the microscope of Ralph, Piggy, and every other boy, it's very easy to go sneaking off into the dark to do what you really wanted to do anyway.
Think of it like sneaking out of the house at night to hang out with your friends. Do you leave through the front door right after supper? Or do you wait until you think mom and dad have fallen asleep watching TV and leave through the egress windows in the basement? These boys chose the windows in the basement.
Jack uses an abrupt approach to trying to get the boys to vote for him as chief. He uses threats and intimidation, rather than reason, and this causes the boys to hesitate. Ralph still has a tenuous hold on the boys and the importance of order and responsibility. Yet, once things calm down, some of the boys decide to go with Jack.
Most likely, the boys feel that it will be more fun and adventurous to be in Jack's group. Ralph's group is focused on creating a civilized society, and this is not nearly as appealing as becoming hunters and running wild.
Remember the Beast??? Jack becomes a leader because he promises that he can protect them from the beast. But the beast in the first place isn;t real and even he knows that, he just manipulates them into thinking it is real. Now think of why everyone mostly slips off to join him. Would you want to be in a group that can't hunt and everything seems strict? or be in a group that can hunt and you can eat meat everyday, also the leader can protect you from the "beast", well of course every one woudl choose the latter group.
The boys do not vote Jack to be leader because Jack does not ask them to. Jack asks, “Who thinks Ralph should not be leader anymore?” The other boys are follows and not leaders, they were never going to be responsible for making such a large decision. Ralph and Jack were the leaders and were making the decisions. The boys that slipped off with Jack later were mostly the same boys that were apart of Jack’s choir when they landed. They were always followers of Jack.