The "ones who walk away from Omelas" are those who have seen the miserable and neglected child and who, evidently, cannot accept its misery in exchange for their own happiness. Children are told about this child when they are between the ages of eight and twelve, whenever they seem ready and capable of understanding. Then, if they wish, they are taken to see the child in its horrible broom closet, sitting in its own excrement and crying out. Some of them simply do not go home after seeing the child, to "weep and rage" as others do. They just head out in the the street and walk out of the city, across the farms, toward the mountains. The narrator says that these ones who walk away "seem to know where they are going." In addition to adolescent boys and girls, there is also sometimes an older man or woman who "falls silent for a day or two, and then leaves home."
The ones who do not walk away seem to justify the child's treatment to themselves:
Their tears at the bitter injustice dry when they begin to perceive the terrible justice of reality, and to accept it.
They tell themselves that the child, were it allowed to come out, is "too degraded and imbecile to know any real joy" because it has lived in fear for too long. Therefore, we can assume that the people who leave are those who cannot justify the continued misery of the child, even is required for them to maintain their own joyful existence.