We know that most of the people who live in Omelas find a way to justify the choice to allow the miserable child in the broom closet to remain there. Though, at first, they cry about the "terrible paradox"—that their happiness and joy can only exist at the expense of this child's awful and permanent misery—they begin to reason that "[the child] would not get much good of its freedom," if it were to be released. They school themselves to accept the "terrible justice of reality," and it seems to render them more "compassion[ate]" as a result; they are "so gentle with children" because of the child. In short, the people who stay are happy.
Therefore, perhaps the people who leave do so alone because they do not feel it is their right to make decisions for other people. They are unwilling to accept joy at the price of another person's misery, but if they were to bring the child up out of the darkness, then it would destroy the happiness of all, without the consent of all. Imagine how bitter the others would be if one person made such a choice. Further, if the ones who leave were to try to convince others of the injustice they, apparently, perceive, then those people would not really be making the right choice for the right reason. Each individual has to make this determination for him or herself: am I willing to accept the abject suffering of another in exchange for my joy?