The village post office in the remote backwater of Ulapur was established by an Englishman, the proprietor of a nearby indigo factory. The suggestion here is that Ulapur, like many small towns and villages in the developing world, is effectively a company town, a place where the main industry exerts a fair measure of political, as well as economic, power.
It would seem from Tagore's description of the place that there's not a whole lot for the inhabitants of Ulapur to do. We're told that the men employed in the indigo factory have no leisure. This would explain why the postmaster finds it impossible to converse with anyone. No one has sufficient leisure to be able to develop their minds and engage in the kind of intellectual pursuits he finds so congenial.
All that the village seems to have is a factory and a post office. The postmaster is trapped in one and doesn't have much time for the men who work in the other. The Englishman who set up the post office may have thought he was doing the local villagers a favor, but in actual fact, he's inadvertently highlighted to the postmaster just how different Ulapur is from his native Calcutta. And not in a good way.