If we were to say "James is a man of 1800," it would mean that he in some way belongs to the specific year of 1800, but not to 1801, 1802, 1824, 1853, and so on.
When, however, we say "James is a man of the 1800s," it means that James belongs to all the years that begin with "18-hundred," that is, to the entire 19th century.
For example, it makes perfect sense to say that Benjamin Franklin was a man of the 1700s--it is actually written without the apostrophe. He lived his entire life during that century, and he was in many ways one of the most outstanding men of the time. It would not make sense, however, to say that he was a man of 1700, because in 1700 he had not yet been born!
The reason why there is an S at the end of the number 1800 is because the sentence is referring COLLECTIVELY as the years that comprised the entire 1800-1899, along with its specific characteristics, social changes, and historical events that came along during that time.
In other words, the addition of the s at the end of the number 1800 would be the grammatical mechanics used to identify that the sentence is referring to the complete one ninety nine years within that period of time, hence, making it a plural that is written without the apostrophe.
Hope this works a little bit.