This is a great question, I think, and often sets off a lively debate. There are many who think that bureaucracy is a terrible hindrance to the functioning of government, particularly those who have run afoul of it for one reason or another, like Yosarian in the famous Catch-22 (Heller). However, as someone who has actually worked for government in a policy-making position, I have seen for myself the value of bureaucracy and come down squarely on the convenience side, along with the prominent sociologist, Max Weber. At the very least, I consider myself to be in good company.
Imagine, if you will, that at the change of an administration every four years, for example, a new president or a new governor, the new president or governor hires all new federal or state employees. Each department and agency will have only new employees. The chaos that would ensue would stop all government functions in their tracks. The attorneys in the Attorney General's office would have to take over cases they had no knowledge of, the people who are responsible for federal housing programs would have a steep learning curve, and the state's program for the aging would take months and months to get back up to speed. No one would have any idea of how to do anything, no one would have any institutional wisdom accrued, and consequently, the government would not be able to perform the myriad tasks for which it is responsible, essential or otherwise.
According to Weber, the bureaucracy solves these problems, and I have personally observed this to be true. If you have a cadre of people who are trained for the job of carrying out the government's policies, that cadre carries on no matter what. There is turnover, to be sure, but less than you might think, usually far less than the turnover in many major corporations. The clerks are on the job, processing papers, fielding phone calls, maintaining websites, and doing all the work of government that could never be done if people started anew every four years. That is the beauty of the bureaucracy. The wheel need not be reinvented. Policy-makers will come and go; policies will change. But the bureaucracy is seldom slowed by this, since much of the work it does is repetitive in nature and changes are usually just incremental. I worked for a state agency in which commissioners came and went, all with their own ideas, but the bureaucracy of the agency, its investigators, secretaries, and IT people went about serving the people all the time, day in and day out.
I would be the first to admit that there are bureaucrats who can be unbelievably annoying, those who blindly carry out tasks without making any allowance for the human element, or sometimes without even logic, and that is not the kind of bureaucracy any government needs, nor is it the kind Weber contemplated. The bureaucracy is a perfect concept, but since it is carried on by human beings, it will remain imperfect. Still, it is so much better than the alternative, which is to start anew in every administrative change and have hundreds of know-nothings who will need two years to get up to speed.