What we must understand is that the term "speech" encompasses far more than what people say. It includes art, film, music, photography, all means of expression in one form or another. So when we talk about free speech, we are talking about far more than the words we utter to one another. The limitation on all forms of expression is the inevitable result of limitations on speech.
With that understanding, I would like to add another perspective on this, which is that if we allow the government to control what can and cannot be said, democracy is destroyed. That might seem like an extreme statement, but history and current events tell us it is true. If we cede this authority, to express ourselves freely, even elected leaders have an opportunity to quash critical commentary, what might be considered subversive literature, political cartoons, and so on. That control assures people in power the wherewithal to remain in power, since anyone opposing people in power can be silenced by this means. One example is Germany before World War II, when Nazis burned books that did not reflect the positions of the regime. Current examples include Egypt, which puts reporters on trial for their reportage and all the Middle Eastern countries which would not permit publication of the Charlie Ebdo cartoon of Mohammed.
Certainly, there are legitimate arguments to be made for some limitation of free speech, as Germany now will not allow the swastika anywhere but a museum and as the United States is beginning to reflect on the use of the Confederate flag. Sexual harassment is not protected speech. The elevated sentencing in hate crimes is largely based upon an accompaniment of hate speech of some sort during the commission of a crime. Commercial speech is protected very little. And child pornography is not protected at all. National security interests are another area in which limitations can be legitimate. If we wish to do a surprise invasion of another country, it is logical for government to want to keep such a plan from being known to the public.
However, since this is such a slippery slope, we want to be very cautious about what rights we give to a government to limit our freedom of speech. It is not melodramatic to say that a government that has this power is a government that we do not want. Probably the very best literary work that provides us with a glimpse of such a future is Bradbury's Fahrenheit 451.
This is, of course, a matter of personal opinion. Let me make an argument for each side of this question and allow you to decide what you think.
I do not think that anyone (or at least hardly anyone) would support the idea of banning books and speech simply because they offend someone on any basis whatsoever. For example, let us say that it offends me if someone says that President Obama is a socialist. There is no way that that sort of speech should be banned.
However, it is easier to argue that speech should be banned if it offends people by insulting them on the basis of race, religion, sex, or other very important personal attribute. Some polities have tried to ban such speech on the idea that it is hate speech. For example, we could argue that it should be illegal to print or say something that argues that non-whites are unintelligent and genetically prone to committing crimes. We could argue that this sort of speech is not meant to persuade. Instead, this sort of speech is the moral equivalent of an assault. When people engage in speech like this, they make it harder for others to lead happy lives in our society. Because the intent of such speech is to harm others, it should not be protected by the First Amendment.
Other people would disagree with this perspective. They would say (as the US Supreme Court has) that the government should not be able to ban speech on the basis of its content. The best argument against banning such speech (in my view) is that it is very difficult to know where “hate speech” ends and political speech begins. For example, what would we do about Donald Trump’s remarks about Mexican immigrants? By saying that they were typically rapists and murderers, Trump deeply insulted them on the basis of their nationality. But isn’t there some degree of a political aspect to that statement? If we ban him from saying that, how can we have a true debate on illegal immigration? It is, in this view, too hard to know where the boundary between hate speech and political arguments lies. Therefore, it is better for us to simply allow all speech. When people say things we do not like, we should fight back by arguing with them, not by denying them the right to speak.
Which of these perspectives makes more sense to you?