One could argue that the theme of the story is the perennial one of man's destruction of his natural environment. Instead of protecting the large hole, treating it as a precious natural phenomenon, the people in this throwaway society see it as nothing more than a giant dumpster. Before long, the hole is being used as a depository for all kinds of junk: nuclear waste, corpses, classified secrets, and lots of other stuff that people want to hide.
The hole could've been put to so many other, more environmentally-friendly, uses. Instead, people saw it as nothing more than a giant landfill. To be sure, a beautiful city soon grows up around the hole, but it's been built on foundations of sand, so to speak. It only exists because of the many ecological horrors committed in the years since the hole was discovered.
The overriding message of the story seems to be that, no matter how hard we try to avoid the consequences of our actions, our destruction of the environment will always come back to haunt us. Maybe not today, maybe not tomorrow, but definitely at some not-too-distant point in the future, as the workman sitting on top of the skyscraper at the end of the story discovers.