For Mill, human freedom was a concept upon which he placed much in way of primacy.  Mill's vision of freedom was fairly pristine, and one where no external intrusion should be present.  Yet, I think that Mill was equally passionate about the idea that one's freedom should be limited when this exercise of freedom is something that is meant to do harm to others:

The sole end for which mankind are warranted, individually or collectively, in interfering with the liberty of action of any of their number, is self-protection. That the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others. His own good, either physical or moral, is not sufficient warrant.

In this light, Mill's vision of freedom can be seen as a construction of two levels.  Individuals possess a right to act on their own autonomy.  At the same time, individuals also possess a right to be left alone.  Mill's freedom seeks to protect both conceptions, ensuring that the former does not step on the latter.  In this light, freedom is complex and is one where positive freedom, the right to act on one's own sense of autonomy, must recognize its natural limits to negative freedom, the right to be left alone.