To me, the major drawback of allowing freedom of speech is that you have to allow very hateful types of speech that can actually harm your society. US courts have consistently ruled that even really horrible forms of speech have to be protected.
The classic case of this is when the Supreme Court ruled that a group of American Nazis had to be allowed to hold a parade through a town in Illinois (Skokie) which had a large population of Jewish Holocaust survivors. This type of speech clearly hurt these people but had to be allowed under freedom of speech. Similarly, courts have held that "hate speech" cannot really be banned, even if it is based on race or other things like that.
So freedom of speech is clearly a two-edged sword. The negative is that it allows people to say things that hurt others and that can undermine the cohesion of a society.
Freedom of speech, can be treated as a power and as with the power to do anything else, it also comes with responsibilities. This freedom has to be used in a responsible manner. It is though, often likened to the power to say anything that a person likes about anything else, without any restraints. This could mean the freedom being used to spread false information about a person to damage his/her reputation, or enticing people to do something which harms them or leads to someone else getting harmed.
The constitution therefore has a few reasonable checks in place to ensure the freedom of speech is used in a responsible way and the consequences are always kept in minds. It is also made sure that the freedom cannot be used to spread false information and harm someone else.