Academic journal article Canadian Journal of Experimental Psychology

Basic Statistics and the Inconsistency of Multiple Comparison Procedures

Academic journal article Canadian Journal of Experimental Psychology

Basic Statistics and the Inconsistency of Multiple Comparison Procedures

Article excerpt

Abstract This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD(5%)). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several wellknown MCPs by focussing on a particular operating characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments. The procedure that behaves best according to this criterion is the unrestricted least significant difference (LSD) procedure. The unrestricted LSD is therefore recommended with the proviso that it be used as a method of generating hypotheses to be tested in subsequent experimentation, not as a method that attempts to simultaneously formulate and test hypotheses.

The format for this paper is as follows. First I shall discuss the various ways in which statistical information can be summarized in scientific papers, and recommend usage of the least significant difference (LSD) as a succinct and useful measure of experimental variability. Then I shall illustrate differences in an important operating characteristic between various multiple comparison procedures by analyzing a specific data set from an experiment with 32 treatments, and subsets corresponding to selected treatments. This leads to a recommendation that planned and unplanned comparisons between treatment means should be made using the same statistical tool (e.g., a 5% level test per comparison), without any adjustment for multiplicity. The proviso is that experimenters should clearly distinguish between the formulation of new hypotheses involving such comparisons (that require confirmation in subsequent experimentation), and the testing of pre-existing, a priori hypotheses that are being confirmed in the current experiment.

Basic Statistical Measures

As an agricultural research statistician attempting to write an article for an unfamiliar readership, I decided to browse through the 2001 volume (No. 55) of the Canadian Journal of Experimental Psychology in order to gain an appreciation of the usual method of presenting statistical information. I especially searched for papers in which data were assumed to follow a "normal" distribution, leading to the calculation of pooled or unpooled standard deviations and related statistical measures. I selected five such papers for copying and further perusal (Baranski & Petrusic, 2001; Christie & Klein, 2001; Gold & Pratt, 2001; Hubbard, 2001; Shore, McLaughlin, & Klein, 2001).

My study of these five papers yielded the following. In all five papers, F values, MSE (Mean Square Error) values and /; values were reported; these summarized the output produced by the analysis of variance technique. In two of the five papers, the SEM (Standard Error of Mean) was presented, in another paper, 95% confidence intervals were presented for each mean, in another paper the PLSD (Protected Least Significant Difference) was presented, and in the fifth paper none of these three measures was presented. I shall now discuss these alternative measures, spell out the links between them, and describe their relationship to the analysis of variance. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.