Good question because J.M. Coetzee changes our ideas of barbarians. Before the novel I understood "barbarian" to refer to uncivilized people, savages. But after the reading the novel, I see that perhaps there is not much difference between savages and civilized.
In this unnamed village, fear of barbarian attacks causes Colonel Joll to seek out savages who live close by and bring them in for questioning and torture. These scenes are quite brutal. Fear of the barbarians causes the supposedly civilized members of the Empire (could it be the British? I think Coetzee is implying that the Empire is comprised of the British), to act in the most inhumane ways toward innocent natives. Their brutality is presented as savage, as the threat of a barbarian attack seems more and more unlikely. The members of the Empire become image of those they deem their enemies.
The Magistrate--the narrator and main character--exploits the natives as well, but he makes the attempt to atone for his past abuses. He befriends a native girl who has been blinded through the interrogation. His intentions toward her are not always innocent, but he does journey to return her to her people, and later faces the wrath of Colonel Joll, an official of the Empire, and endures torture for his disloyalty.
In a way the theme is similar to Conrad's Heart of Darkness. We are all savage underneath. In this work, fear of outsiders reduces the civilized to savage, while power leads to exploitation and excesses.
The barbarians are anyone and everyone who is inhumane, brutal, and exploitive.