The logical answer to this question is that freedoms of speech and expression end where other people's begin, like most other rights. Yet determining when the public has an interest in limiting speech has been a major chore for courts in the United States. A few landmark decisions by the US Supreme Court might help illuminate where these boundaries lie.
One of the most famous articulations of the boundaries of free speech was issued by Oliver Wendell Holmes. Writing the majority opinion in the 1919 decision Schenck v. United States, Holmes argued that "the most stringent protection of free speech would not protect a man in falsely shouting fire in a theater and causing a panic." In other words, when speech poses a danger to the safety of others, it is permissable to limit it.
A form of expression that is not protected by the First Amendment is obscenity, but the courts have struggled to determine how obscenity might be identified. The old test established by the Court in Miller v. California was that obscene material could have no scientific or cultural value, but would only be intended to appeal to the "prurient interest." This test has been seriously weakened in recent decisions, however.
Hate speech that targets particular individuals has generally been ruled as protected under the First Amendment, but speech that might be construed as "fighting words," such as encouraging violence against individuals, or, as in a famous 2002 case, distributing the names and addresses of the family members of abortion doctors, can be subject to "strict scrutiny."
Basically, American jurisprudence errs on the side of free speech and expression, but the restrictions that are allowed are those that are intended to protect the safety of others.