In the story “By the Waters of Babylon,” Babylon is not a place that is in the actual story. The title is just an allusion. It alludes to Psalm 137 in the Bible. It is a reference to the idea of longing for something that has been lost.
Psalm 137 in the Bible begins
By the rivers of Babylon, there we sat down, yea, we wept, when we remembered Zion.
This is a psalm of lament, spoken from the perspective of the people of Israel. The psalm was written about the time when the Israelites had been defeated in war by Babylon and had been removed from their homeland and taken to Babylon, the home of their conquerors. The Israelites had believed that God had promised them their homeland (Zion) and that they had now been forcibly removed from this promised land. The psalm details their sadness and their sense of being lost now that this has happened.
We can draw an analogy between the situation of the Hill People in this story and that of the Israelites in the Bible. The Hill People’s ancestors once lived in a promised land of sorts. Their promised land was the one where there was civilization and technology and all of the things that make life easy in modern society. Their promised land was taken away, apparently by a terrible war that destroyed their civilization. Now, the Hill People (and the Forest People, for that matter) have been “exiled” to a much more primitive existence. By the end of the story, John realizes what his people have lost. He laments that they have lost it as the Biblical psalmist laments the loss of the promised land.
Thus, Babylon in this story does not refer to a place that is in the story. Instead, the word appears in the title (and only in the title) to allude to Psalm 137 and to set out the idea of regret for things that have been lost.