According to Mill, freedom of expression should be nearly absolute. He believes that the only reason to infringe on the right of freedom of expression is to prevent harm to other people. A major problem with his argument, however, is that he fails to define harm.
A conservative critique of Mill's theory would likely focus on the idea that there are kinds of expression that are very harmful to a society. They would focus on things like pornography that they see as hurting the fabric of society.
I would point out that this is not really a liberal-conservative debate. Liberals are also likely to try to suppress speech that they dislike (like hate speech) even if it would not violate Mill's harm principle.
To me, the good part of Mill's argument is that it tends to defend as much speech as possible. However, it does not really help us understand what speech we should protect since it does not help us understand what speech is harmful and what is not.