Without a doubt, the supervisory duties that I least prefer are those that bring conflict and unhappiness. These are the sorts of burdens that make positions of authority much less pleasant at times than positions in which I have had no supervisory duties.
One thing that a supervisor has to do is to admonish or even discipline workers who are not doing well. This can be very stressful because it is fundamentally unpleasant to tell someone that they are doing badly. It is also fundamentally unpleasant to, in essence, threaten them with sanctions. This gets to be even worse in situations where you have actually punish, or even discharge, an employee. These are the least appealing aspects of supervision because they are very emotionally stressful.
I like to help people and to analyze systems. I do not like to make people feel terrible. For these reasons, the duties I least prefer are the disciplinary duties that go along with being in a supervisory position.