"The Great Wall of China" is not a conventional story, with a clear sense of narrative or plot progression. Rather, it is written as the musings of a nameless narrator who took part in the building of the Great Wall of China.

The story begins by detailing the construction method of the Great Wall itself, which was built piecemeal by many different crews working independently of one another. The narrator reveals himself to have been a supervisor to one of these work crews. He muses on the ways in which he, and men much like him, draw meaning from their part in the creation of the Great Wall, and he suggests that the wall's particular method of construction ensured that enthusiasm could be preserved, even in the wake of that project's overwhelming scale. He recalls how, when the task of building any section of the wall was completed, the supervisors were rewarded with honors and returned to their villages, where gradually their enthusiasm recovered and they set off once more to continue work on the wall.

The subject of the Great Wall is only Kafka's starting point, and from here the narrative turns toward a more abstract analysis. As we learn, the unnamed narrator comes from Southern China, and thus the construction of the wall can never personally affect him—the villages in the south are in no danger from northern raiders, because they are far removed geographically. Similarly, a general sense of abstract incomprehensibility emerges in the narrator's story concerning the emperors, the Imperial Court, and his own immediate superiors. For the narrator, these people are too far removed from his own life and experience to possess any sense of reality to him. Thus, the real, living emperor of China is insignificant next to the idea of the emperor, because the vast majority of people in China can never know the emperor as a flesh-and-blood person—they can only access the idea of him. Thus Kafka's analysis shifts from the level of concrete experience, as represented in the building of the Great Wall, to levels far more abstract.