What Auden is doing here is to emphasize the relative unimportance of the ordinary individual in relation to the modern state. The modern, bureaucratic state as presented by Auden thinks of itself as incredibly important and that overriding sense of self-importance is reflected in the use of capital letters to refer to the various institutions of state.

The recently deceased citizen is known only by a number, a number by which he was identified throughout his whole life by the state. This process of dehumanization was implemented by organizations such as the Union and the Bureau of Statistics the better to control the unknown citizen and countless others like him. The identity of such organs of state is clear for everyone to see, expressed in grand capital letters to emphasize their immense power and prestige. But the unknown citizen's identity as an individual human being was erased long before he passed away. Throughout his long and undistinguished life he became nothing more than a cog in a gigantic machine, manipulated to serve some higher end by the powers that be, ends whose own importance is also reflected in capital letters. (Such as "The Greater Community," for example). Ironically, it is only in death that The Unknown Citizen is deemed sufficiently important to receive his very own set of capital letters.

If you use proper grammar, you Capitalize Proper Nouns.  The corner is a place, it doesn't need to be capitalized, but Mount Rushmore is a place, and it does need to be Capitalized because Mount Rushmore is a proper noun. 

In the Unknown Citizen, Auden is using Press, Union and Public Opinion as proper nouns.  The Press is a specific person in the story (i.e. a proper noun).  The Union is another specific person (i.e. a proper noun). 

Why he makes these proper nouns, relates to the message he is trying to illustrate with the story.  I hope that helps you understand his marriage.