By the Waters of Babylon is a short story which deals with a post world war civilization. The characters we meet are probably descendants of the survivors. The "temples" and the "caves and tunnels" are skyscrapers and subways that have remained over the long years since the destruction of civilization.
Temples = Skyscrapers
Caves annd Tunnles = Subways