While this does make some sense, there are a number of ways to disagree with the statement. Let us look at two of the more important ones.
First, even if we do not do anything wrong, we do not want to live in a police state where we are always under surveillance. It simply degrades our quality of life.
Second, even if we do not do anything that is legally wrong, we might do things that we would not want our neighbors to know about. For example, a person who watches pornography would probably not want his or her neighbors, co-workers, and employers to know about his perfectly legal actions.