# Please help with these research questions in regard to the study below: Describe the research question for this experiment. What were the null and alternative hypotheses? Were the results...

• Describe the research question for this experiment.
• What were the null and alternative hypotheses?
• Were the results of this test statistically significant?
• If so, why were they significant?
• Would the researchers reject or fail to reject the null hypothesis?
• Do the results provide sufficient evidence to support the alternative hypothesis?
• What are some possible limitations to this study?
• Describe the difference between practical and statistical significance.

A group of researchers conducted an experiment to determine which vaccine is more effective for preventing getting the flu. They tested two different types of vaccines: a shot and a nasal spray. To test the effectiveness, 1000 participants were randomly selected with 500 people getting the shot and 500 the nasal spray. Of the 500 people were treated with the shot, 80 developed the flu and 420 did not. Of the people who were treated with the nasal spray, 120 people developed the flu and 380 did not. The level of significance was set at .05. The proportion of people who were treated with the shot who developed the flu = .16, and the proportion of the people who were treated with the nasal spray was .24. The calculated p value = .0008.

embizze | Certified Educator

(1) The null hypothesis is that there is no difference between the treatments; the alternative hypothesis is that there is some difference.

Since p=.0008 is less than the level of significance alpha=.05, so the results are statistically significant.

(2) The researchers would reject the null hypothesis. If the null hypothesis were true, a sample with results like this would only occur 8 out of 10000 trials. Since this is highly unlikely ( and far below the 5 in a hundred level of significance required) we would believe that the null hypothesis was incorrect.

(3) The sample is large enough. We are not told what the population is. (All US adults, US adults who are taking a vaccine, etc...) Assuming the samples are truly randomly chosen from a well-defined population the sample seems adequate.

(4) Possible limitations include an improperly defined population or the selection of the sample not being truly random. Also, the experiment could not be double-blind (it is hard to disguise whether you are getting a shot or the mist.) There is no control group to define the likelihood of an unvaccinated individual contracting the flu, though this might be known for the given population. Was the type of influenza considered? (Were the test subjects exposed to a particular strain or were they tracked during the flu season and only reported if they got the flu?)

(5) First identify any problems in the first test (improper population, etc...) and fix those. Make sure to control for any confounding variables including flu strain, age and health of participants. Insure the experiment is double-blind by including a control group who are issued a placebo (salt-water spray or inert shot) to reduce the placebo/nocebo effects or give everyone two treatments-- one placebo and the other that actual treatment. That is, if I am scheduled to get the shot, I also get a saline mist, where those destined to get the mist also get an inert shot.

(6) Assuming that the study results are correct it appears that receiving the vaccine as a shot is more effective than through nasal application. But we are not told what the percentage of untreated people who would contract the flu is. Suppose 1/4 of untreated people exposed to the flu actually become infected. (So 125 out of 500.) Then the small advantage of nasal protection might not be worth the time and expense. Indeed, while the difference between shot and mist might be statistically significant, the results might not be practically significant in that the cost (time, money, pain) might not outweigh the benefit (reduced chance of getting the flu.) In the case of immunosuppressed patients the risks of getting the flu are high, but for most healthy people the flu is typically an irritation so a small reduction in risk is not practically significant. (If the vaccine virtually guaranteed no infection this would be different.)

txmedteach | Certified Educator

1) Null hypothesis and alternative hypothesis have specific definitions. The null hypothesis posits that the treatments do not differ from each other. The alternative hypothesis posits that the treatment results in different incidences of the disease. Significance is judged by the level of significance (the theoretic probability that the difference you are seeing is due simply to chance—here it is 0.05). This result is statistically significant because the p value (measured probability of the difference between the two tests being due to chance) is less than the level of significance.

2) Because the p value is less than the level of significance, the researchers would reject the null hypothesis.

3) On its face, the results appear to plainly support the injectable vaccine; however, we are making multiple assumptions by doing so. Almost every manuscript that reports results like this endeavor to perform some analysis that shows how the two measured populations were similar and how they differed significantly. We are not given the information in this case. Without having this information, it is difficult to comment on the result.

4) Limitations are best assessed by looking at the gold standard for medical research: the randomized, double-blinded, controlled trial. We are not told the subjects were randomized. As the previous commenter pointed out, patients cannot be blinded to their intervention, though the physician may be (you don't need to be a doctor to deliver a medication). Finally, there is not a control group. Ideally, you would be able to compare the interventions to a population of people that were not treated as a control. Finally, you would want to minimize local effects on your studied population. Normally, this is done by using a multi-centered trial, in which patients from multiple institutions are examined. However, we are not told that this is the case.

5) One thing you would want to change for a follow-up study would be to change the things you note as limitations. You would also want to examine the current population to see if there are long-term effects from the interventions.

6) Statistical versus practical significance is one of the more important questions in many medical studies.  This is where effect size is important. For example, there might be a statistically significant difference in neurologic recovery after receiving high dose steroids quickly after a spinal cord injury. However, the clinical effect was judged to be small enough that it did not outweigh the comorbidities associated with high dose steroids in these severely injured patients (namely, pneumonia), and high dose steroids were no longer recommended as of 2013.

liajferguson | Student
• The null hypothesis is that there is no significant difference between the effects of the two treatments. In other words, the shots and the spray would not affect the proportion of people who developed the flu. The alternative hypothesis, therefore, is that there is a significant difference between the effects.
• The calculated p-value, 0.0008, is less than the alpha value, 0.05, so the results of this test were significant.
• They were significant because, in the two populations, the proportions of those who developed the flu were different enough to only happen at random 0.08 % of the time, which is less than the chosen alpha value of 5%.
• The researchers would reject the null hypothesis.
• The results provide sufficient evidence to support the alternative hypothesis.
• The sample size is large enough to give statistically significant results, and the participants were chosen at random, so the sample should be adequate to represent the population from which the participants were chosen.
• Possible limitations include absence of a control group and a placebo group. The study was also not blind because the patients knew which treatment they were being given, nor was it double blind because the doctors knew which treatment they were administering.
• As a follow up study, I would split the participants into four groups, control, placebo, nasal spray, and shot. And I would make it double blind meaning each participant in the placebo, nasal spray and shot groups would get both shot and spray administered, one or both of which would be a placebo. The person administering the treatment would also not know which was medicinal and which was a placebo.
• Statistical significance is only a measure of the likelihood that results were achieved at random, not a measure of the usefulness of those results. With enough participants in a study, you may find that the likelihood of contracting the flu using only nasal spray was 1% more likely than with the shot and get a p-value of .04, but if, for instance, the spray was cheaper, more easily dispersed and could ultimately prevent more illness, that study would have little practical significance.