The phrase "by the waters of Babylon" is an allusion to Psalm 137, in which the Israelites mourn their exile from Jerusalem and weep over their memory of their lost homeland. The waters of Babylon are not where they belong. They long to be back in Israel.
Likewise, John would like to return to the former glory of the civilization he realizes his people were once a part of. When he travels to the Place of the Gods, which we as readers recognize as New York City, he realizes it was built by human beings like his people, not by Gods. Like the Israelites, he laments over what was lost and holds the hope his people can return to it.
The title is an example of dramatic irony, which occurs when the reader knows what characters in a story do not. Because they have lost so much knowledge, the Hill People cannot know of their heritage in the Judeo-Christian tradition, just as they cannot recognize the Place of the Gods as Manhattan.
John contrasts his own people, the Hill People, to the Forest People. His own people have made advances the Forest People have not. As John notes:
We are not ignorant like the Forest People—our women spin wool on the wheel, our priests wear a white robe. We do not eat grubs from the trees, we have not forgotten the old writings, although they are hard to understand.
The Hill People are better positioned to begin relearning the old knowledge. The title is important because it suggests the hope that, like the Israelites, the Hill People will return one day to the Promised Land of their former civilization.