Academic journal article The Journal of Parapsychology

A Meta-Analytic Comparison of the Sensitivity of Direct Hits and Sums of Ranks as Outcome Measures for Free-Responses Studies

Academic journal article The Journal of Parapsychology

A Meta-Analytic Comparison of the Sensitivity of Direct Hits and Sums of Ranks as Outcome Measures for Free-Responses Studies

Article excerpt

In a typical open-deck free-response extrasensory perception study, a target is randomly selected from a set of four pictures. The picture is kept hidden from the receiver, who then attempts to describe its content. The receiver is then shown a duplicate set of the four pictures (to avoid handling cues). Still not knowing which is the target, the receiver (or an independent judge who is blind to the target's identity) rates or ranks the four pictures in terms of how well each corresponds to the earlier attempted description. This procedure is repeated on other trials until the study is completed.

In recent years, two measures - direct hits and ranks - have become the most popular ways of representing such a study's outcome.(1) A trial obtains a direct hit if the target picture obtains the highest rating or rank, and otherwise is counted a miss. The study's outcome is assessed by means of the binomial distribution, with p = 1/4. If, instead, ranks are being used as the study's outcome measure, then the rank assigned to the target is the trial's outcome and the study's outcome can be determined by comparing the sum of ranks across all the trials to the chance expected distribution (Solfvin, Kelly, & Burdick, 1978).

Although researchers, in effect, find themselves faced with a choice between direct hits and sums of ranks, there are no clear indications at present as to which - if either - of the two is the more sensitive measure. However, it is important that if one measure is superior to another then that measure should be used because effect sizes in free-response ESP studies are very low. In 1986, Utts used the then current estimate of ganzfeld ESP study effect size of a hit rate of 33% (MCE being 25%) to calculate that over 100 trials would be needed to give a ganzfeld study a 50:50 chance of reaching statistical significance (Utts, 1986). Recent research indicates that ganzfeld studies conducted after Utts' calculations are not achieving even that effect size (Milton & Wiseman, 1997). It clearly makes sense for researchers to be sure that they are using the most sensitive measure of psi available in order to achieve the maximum statistical power with the minimum of time-consuming and expensive trials.

As far as theoretical arguments about which measure should be better are concerned, no clear winner emerges. Ranks might be expected to be the more sensitive measure of ESP performance because they use more information from the trial than do judgements of direct hits, which do not distinguish between near misses and clear failures. However, as Hansen and Utts (1987) point out, whether sums of ranks are in practice more sensitive than direct hits depends both upon how ESP manifests itself and on how judges behave when assigning ranks. If psi operates in an all-or-nothing fashion in each trial, so that it either operates so strongly that a direct hit is certain, or does not operate at all so that each rank is equally likely, then direct hits, not ranks, might be the most sensitive measure. If, instead, psi operates like a weak signal hidden in noise, then the frequency distribution of ranks should tend to shift toward the lower-rank, above-chance end and ranks will be the more sensitive measure. However, in either case, the judge's behavior can affect which measure should be best. Even if psi acts in an all-or-nothing manner, a judge who finds it hard to discriminate which judging-set item best matches the receiver's response might often turn a direct hit into a second, or, less frequently, even a third rank, in which case ranks would once again be the more sensitive measure. This situation could easily arise if judging sets contain items that are not sufficiently different from each other. Conversely, even if psi acts as a signal buried in noise, a judge who pays attention only to accurately assigning a rank of 1 to the item that best matches the response and ignores the other judging-set items, assigning their ranks more or less at random, will make direct hits the more sensitive measure. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.