It was common practice among the upper classes in those days for women to leave the dining room after dinner while the men smoked cigars and engaged in the kind of conversation traditionally reserved for the male of the species. More often than not, such conversation would revolve around subjects, such as politics, that were not deemed appropriate for ladies of sensibility.
In this particular case, it's not just the ladies who withdraw after dinner; Mr. Wharton and his son have to take their leave, too, on account of the death of a near neighbor. Even allowing for tradition, it's likely that the ladies would still have left anyway, as it's clear that an argument over the American Revolution is brewing between two of the guests: Dr. Sitgreave, a surgeon who supports the rebels, and Colonel Wellmere, a British army officer, who, for obvious reasons, does not.
The two men soon get into a heated argument over the nature of liberty. Wellmere chides Sitgreave over the colonists' hypocrisy in demanding liberty from the British while denying it to their slaves. Sitgreave counters by arguing that, in due course, slavery will eventually be abolished in America, but only after the colonies achieve independence from the mother country.