Statistics teaching should be integrated with empirical substance. Statistics is not varnish, to be applied after the experiment is done. Statistics is an integral component of the research plan—beginning with choice of problem for investigation. Statistics is important at every intermediate level, including apparatus and procedure, and concluding in the interpretation of the results.
Students almost inevitably come to view the significance test as the be-all and end-all. This test condenses the entire study into a single number of pivotal importance for making claims about what the study shows. A positive answer to the question, “Is it significant?” thus comes to be considered the ultimate goal. “It is significant!” seems all the more potent because “significant” carries undertones of everyday meaning.
The essential question, of course, is what the results mean. The significance test has the necessary—but minor—function of providing evidence of whether there is a result to interpret. What this result may mean depends on considerations at deeper levels.
The more important functions of statistics are in these deeper levels. These more important functions apply at the planning stage, before the data are collected. These more important functions condition the meaning and interpretation of any result that may be obtained. These more important functions of statistics need to be understood in organic relation to the substantive inquiry.
The significance test, in contrast, applies after the data have been collected. It is then too late to correct missed opportunities and shortcomings in the research plan. Finding a significant decrease in felt pain may not be worth much if the placebo control was overlooked. The placebo control may not be valid if the treatment was not “blind.” And even the most careful procedure may founder if you or your research assistant stumbled with the random assignment. These three examples illustrate vital functions of statistics that operate in the planning stage, before the data are collected. This is where statistics is most needed—and most effective.