Rebecca Zwick Educational Testing Service
Research in the behavioral and health sciences frequently involves the application of one-factor analysis of variance (ANOVA) models. The goal may be to compare several independent groups of subjects on a quantitative dependent variable or, alternatively, to compare measurements made on different occasions or under different conditions on a single group of subjects. If there is reason to believe there are differences among the groups (or occasions or conditions), the researcher frequently wishes to compare the means in a pairwise fashion. Although the procedures for conducting omnibus hypothesis tests for one-factor ANOVA models are familiar to most researchers, the issues that must be considered in choosing pairwise multiple comparison procedures (MCPs) are not as well-understood. In this chapter, the selection of pairwise MCPs for one-factor ANOVA models is considered, following a discussion of Type I error and power issues as they apply to the testing of multiple hypotheses. Although the focus is on the independent-sample case, repeated measures models are considered briefly as well.
Any student who has taken an elementary statistics course can recite the definitions of Type I error and power: The Type I error rate is the probability of rejecting the null hypothesis when the null hypothesis is true and power is the probability of rejecting the null hypothesis when the null hypothesis is false. However, these concepts become much more complex when applied to multiple hypothesis tests, such as MCPs. In the multiple-comparison case, it is possible to