Can Meta-Analysis Resolve the ESP Controversy?
Meta-analysis is a quantitative, analytical approach to the review of scientific literature (Green and Hall 1984). The initial applications to psychology were in major areas of controversy, such as employment test validity (Schmidt and Hunter 1977), experimenter expectancy effects (Rosenthal and Rubin 1978), sex differences in conformity (Cooper 1979), and effects of psychotherapy (Smith and Glass 1977). Meta-analysis has become a popular technique for the exploration of many areas where experimental outcomes are in conflict, including parapsychology (Honorton 1985; Hyman 1985).
Meta-analytic methods do have some potential for resolving parapsychological controversies. However, I will argue that there are serious limitations to what such methods can accomplish with old data—data on which skeptics and believers disagree vehemently. It is apparent that skeptics and believers do not agree on the quality of parapsychological studies (and hence on how much weight each study should receive in the analysis). If this is true, then meta‐ analysis may be premature. It may be preferable to collect new data, under conditions prespecified by both parties to the debate. With research standards prespecified, the problem inherent to meta-analysis, of subjective bias in the coding of research quality, could be avoided.
Before exploring the issue of quality coding, and possible resolution of the issue, a brief account of meta-analytic methods is in order. The initial step in meta-analysis (after delineation of the problem area) is a definition of the data base. This involves setting explicit criteria for which studies should be included in the literature review and which should be excluded. The reviewer then defines the nature of the independent and dependent variables he wishes to consider. Methodological or design quality features can and should be included. All such study characteristics are then coded and assigned numerical values (if only for____________________