The things that John is really seeing are remnants of the once-bustling city of New York. We know that John crosses the Hudson to get to a city with tall buildings, and before even seeing the city itself, John tells readers that he left the god-roads because they are not easy to travel on.
. . . we do not use the god-roads now for they are falling apart into great blocks of stone, and the forest is safer going.
That sentence is a good indicator that the god-roads are paved concrete or asphalt roads. Dirt roads get rutted. They do not fall apart into big blocks. Old concrete does break into large blocks that resemble stone.
As John approaches the actual city, he mentions that he sees god-roads that stretch across the Hudson. He says that they are also broken roads.
I saw that once there had been god-roads across it, though now they were broken and fallen like broken vines.
God-roads that reach across rivers are bridges. New York has a lot of bridges that access it. Many of them are suspension bridges, which is why I believe that John specifically mentions vines in the above quote. Broken and strewn steel cables could look like broken vines.
As for the tunnels and caves, John is talking about the New York City subway system. One good piece of evidence for this is the "temple" that has all kinds of stars across the ceiling.
A mighty temple it must have been, for the roof was painted like the sky at night with its stars — that much I could see, though the colors were faint and dim. It went down into great caves and tunnels—perhaps they kept their slaves there.
The building that he is referring to is Grand Central Station. Its ceiling is decorated with stars and other celestial items. It descends a long way and leads to a plethora of train and subway tunnels.