Teachers in my experience have had one of two issues: classroom management, and administrative management. Having a struggling student is troubling, and when that student sees the only positive attention coming his way is from other students when he misbehaves, he will misbehave without regard to consequences. Often the student does not even see consequences as bad! We are always taught about ways to deal with the struggling student, and many of us find ways to do this. It often takes a great deal of effort, and those of us that succeed should be praised! However, for many other teachers, the constant demand for help and attention required by multiple students exceeds what that teacher is able to give, and that, for sure, will be a hit to confidence.
The other realm is a surefire confidence-killer. If the administration of a school is incompetent, inexperienced, or makes self-serving decisions, the powerless teacher will certainly be affected by being put in a position of quitting (and not being able to find a job for some time AND having to deal with a negative reference) or just doing as told and feeling no different than a teenager at McDonalds being told to man the fry machine. Having leaders is necessary to keep cohesiveness at work and keeping everyone accountable. However, when you have a principal who has no experience in your area telling you you're teaching incorrectly, it can get annoying very quickly. The confidence-killer comes when you realize this person is your boss, can tell you what to do regardless of whether or not a decision is right or informed, and makes more than you will as long as you remain a teacher.
Granted, in Texas (where I taught) there are no unions..and I did teach at a charter school. Maybe I brought the administrative problems on myself! ;-)