The answer to this question can actually be found in Chapter 3, which is when Ralph reveals the reason why the first hut was built by everybody, but the two subsequent huts were only built by a few, and as a result are rather shoddy and in danger of collapsing. The boys just run off and play, swim or eat. Jack does not help with the shelters because he is so desperate to kill a pig, and so he and his hunters do not help. Note what Ralph says about the fickle nature of the boys:

I bet if I blew the conch this minute, they'd come running. Then we'd be, you know, very solemn, and someone would say we ought to build a jet, or a submarine, or a TV set. When the meeting was over they'd work for five minutes then wander off or go hunting.

The reason therefore why the huts were not built was because the boys are unable to stick at any task for a decent length of time. They become distracted and go off and do their own thing. Ralph realises that although he has been appointed leader, he is leader only of a group of boys who are unable to follow through the orders that he gives.