Describe the relationship between levels of confidence and statistical significance and how they affect the results of the data analysis?
A result in data analysis is said to be statistically significant if we are 95% confident, say, that the effect we have observed through analysis is a real one and not one that is manifested by chance alone.
In a statistical test/hypothesis test of an effect of interest we choose a nominal type I error rate alpha. We compare the test statistic to the 100(1-alpha)% cutoff of the sampling distribution of the test statistic and if it exceeds it then we say that the result is statistically significant at the 100(1-alpha)% level of confidence. alpha = 0.05 is a common choice in practice.
By choosing a value for alpha we set an upper bound for the (type I) error we wish to tolerate in our conclusions. Running the scenario over and over again, where the data come from some assumed statistical distribution, our conclusion will be founded 100(1-alpha)% of the time, which we judge to be high enough. We can be 100(1-alpha)% confident that are conclusion is founded based on assumptions we have made about how the data arise and the value of alpha we have chosen.
If we require the level of confidenceto be high then statistically significant results are going to be less frequent upon repeated testing if the result is unfounded. Even if the result is founded, they will be less frequent though perhaps negligibly so. Not establishing a result when a positive result would be founded - a false negative - is a type II error. If the level of confidence specified is low then we will obtain more statistically significant results upon repeated testing, but at the cost of them being more likely to be false positives.
If a result is said to be significantly significant we judge that it is founded on there being a true effect. This may or may not be true, as statistical data analyses are always subject to error.
If the level of confidence is high, a result is less likely to be statistically significant, especially if it is unfounded. If the level of confidence is low, a result is more likely to be statistically significant but at the cost of being a false positive.