I am not that good at figuring out those kinds of hints and details early in a story. Twists always get me by surprise. "By the Waters of Babylon" was not that much different for me either. Looking back at the story, there were plenty of hints that the Place of the Gods was a former human city, but I did not really start questioning what John was exploring until portions of English words started appearing on structures that he was looking at.
I can read letters but I could not understand these. They said UBTREAS.
I know that prior to that quote, the story gave information about really tall structures and "god roads" that extended over the water. I realize now that those things were city streets and bridges, but I didn't have that "wait a minute" moment until I saw English lettering.