The people who leave Omelas are citizens of the town who cannot accept that their happiness is based on the child's suffering.
The people who leave Omleas are regular townspeople. They range from being "one of the adolescent girls or boys" to "a man or woman much older." They have seen the child whose suffering is the reason behind their happiness:
They know that if the wretched one were not there sniveling in the dark, the other one, the flute-player, could make no joyful music as the young riders line up in their beauty for the race in the sunlight of the first morning of summer.
The difference between these people and those who stay is that what they have seen has left a mark that cannot be washed away.
The narrator says these people simply leave. They have left because they know the truth about living in Omelas. They make no attempt to retrieve the child or fight for the child's rights. Rather, the narrator says they simply "go out into the street, and walk down the street alone." They leave the city, walking to an unknown place that is "ahead in the darkness and they do not come back." Their destination is unclear. However, the ones who walk away know where they are going, even if the narrator and us do not.