A useful way to look at data is in terms of variance: If the differences between our treatment means are large—compared to individual differences within each treatment condition—this is a sign of real treatment effects.
This intuitive decision process is made precise in this chapter through analysis of variance (Anova). Formulas are presented for the variance between treatment means (MSbetween) and for the variance of the individual responses within a single treatment (MSwithin). The ratio, F = MSbetween/MSwithin, becomes our decision guide. A larger F is stronger evidence for real differences between our treatments. If F is “large enough, ” we have a statistically significant result, provisional evidence for real treatment effects.
Anova makes “large enough” precise. In terms of Chapter 1, your F ratio is an index of reliability, that is, the reliability of the observed differences between the treatment means.
In practice, a significance test is easy. Just give your data to the computer. It will calculate your F ratio and tell you whether it is “large enough.”
The F test applies to two or more conditions; it includes the t test as a special case. Further, MSwithin = s2, which may be used to construct confidence intervals using the expressions in Chapter 2. Confidence intervals can be very helpful to your reader when you describe your data.
Anova depends on certain assumptions. Two of these, normal distribution and equal variance, are not usually problematic in experimental studies. Independence is critical, but can usually be ensured through careful procedure and random assignment. Practical aspects of How to Randomize are discussed in the appendix beginning on page 77.
It cannot be emphasized too much that the statistical significance test says nothing whatever about substantive significance of your results. The significance test merely tells whether your result has some minimum degree of reliability. This is a minimum first step; unless the result is reliable, there is little point in worrying what it might mean.
Questions of meaning, however, are primarily extrastatistical. Questions of meaning involve considerations at lower levels of the Experimental Pyramid of Chapter 1. Statistics can help with some of these questions, as will be seen in later chapters, but this help requires going beyond the significance test to issues of experimental design.