The barbarian problem is defined by Colonel Joll, who represents the interests of the empire. The "problem" is the perceived likelihood of a barbarian attack on the outposts of the empire along the frontier.
Early in the novel, Joll shares his mission with the magistrate, telling him that the decision-makers in the empire have found reason to worry about an attack. He comes to the frontier to investigate this possibility, but his actions suggest that he may merely be looking for corroboration.
"After killing the grandfather, [Joll] continues the torture until the boy confesses that his people, the barbarians, are preparing an attack against the empire" (eNotes)
The empire may, in fact, want to cultivate an excuse to attack the barbarians (perhaps in order to extend the territory of the empire, perhaps in order to eliminate a nagging political concern, perhaps simply as an expression of power).
The dark glasses Joll wears help to symbolize the idea that Joll only sees what he wants to see. While the magistrate can see no evidence of any barbarian attack and believes that the confessions Joll has extracted are coerced and false, Joll believes all the evidence he finds to support his initial theory.
Thus the barbarian problem is a theoretical problem, most likely concocted entirely by the empire. There is, of course, some chance that the barbarians actually are planning an attack, but there is no evidence of this. The barbarians are effectively silent and absent in the narrative, functioning as a cipher or blank slate that is either unknown and unknowable or, instead, given imbued with qualities that are projected by others.
At times, the magistrate comments on certain issues with the barbarians who seasonally pass through the town. There are "barbarian problems" in this context on a minor scale.
However, the larger barbarian problem at the heart of the narrative has little to do with the actual barbarians and more to do with the perceptions and attitudes of the imperial powers represented by Joll in his cruelty and his mypoic insistence on his own authority/rightness and by the magistrate in his troubled fascination with the inscrutable barbarian girl and what she represents about his position in the world and his complicity in the workings of the empire.