What is wrong with this survey?Seventy-two percent of American squeeze their toothpaste tube from the top. This and other not-serious findings are included in The First Really Important Survey of...
What is wrong with this survey?
Seventy-two percent of American squeeze their toothpaste tube from the top. This and other not-serious findings are included in The First Really Important Survey of American habits. Those results are based on 700 responses from the 25,000 questionnatires that were mailed
The actual type of poll you are referring to in this hypothetical is called a "straw poll", and in polling organizations it is generally believed to be the least reliable kind of survey there is. We see these kinds of polls online and in magazines all of the time.
On a sports website, for example, they may have a poll to see who you think is going to win the Superbowl this year. The results, just like in the survey you mention, cannot accurately measure the population in any meaningful way. It is based only on those who happen to see or read the questionnaire or poll, those who choose to fill it out, and then those who actually choose to mail it in. This is not going to be a representative group of people. Even if it were, as pohnpei points out, you wouldn't know who or what they represent, because no other information about the respondents was gathered. Was it 72% of all who responded? Of women? Senior citizens? The details of such demographics matter when determining how accurate a poll can be.
What is wrong with this survey is that it is in no way random. Therefore, the results returned may not reflect what people in the country as a whole do with their toothpaste.
Let us assume that the 25,000 surveys that were sent out were properly randomized (by race, class, sex, age, etc). Even if this is true, that tells us nothing about the 700 people who returned their surveys. It is likely that people who returned their surveys were in some way different than those who did not. My guess is that they would be more well off and more educated.
A survey can only be completely legitimate if its respondents are random. This survey cannot guarantee that.