Decision Noise: An Explanation for Observed Violations of Signal Detection Theory
Mueller, Shane T., Weidemann, Christoph T., Psychonomic Bulletin & Review
In signal detection theory (SDT), responses are governed by perceptual noise and a flexible decision criterion. Recent criticisms of SDT (see, e.g., Balakrishnan, 1999) have identified violations of its assumptions, and researchers have suggested that SDT fundamentally misrepresents perceptual and decision processes. We hypothesize that, instead, these violations of SDT stem from decision noise: the inability to use deterministic response criteria. In order to investigate this hypothesis, we present a simple extension of SDT-the decision noise model-with which we demonstrate that shifts in a decision criterion can be masked by decision noise. In addition, we propose a new statistic that can help identify whether the violations of SDT stem from perceptual or from decision processes. The results of a stimulus classification experiment-together with model fits to past experiments-show that decision noise substantially affects performance. These findings suggest that decision noise is important across a wide range of tasks and needs to be better understood in order to accurately measure perceptual processes.
Signal detection theory (SDT) has become a prominent and useful tool for analyzing performance across a wide spectrum of psychological tasks, from single-cell recordings and perceptual discrimination to high-level categorization, medical decision making, and memory tasks. The utility of SDT comes from its clear and simple account of how detection or classification performance can be translated into psychological quantities, such as sensitivity and bias. Whether its use is appropriate for a specific application depends on a number of underlying assumptions, and even though these assumptions are rarely tested, SDT has proved useful enough that it is considered one of the great successes of cognitive psychology. Yet, SDT has also undergone criticism, which began to emerge when this theory was relatively young.
Criticisms of SDT
SDT assumes that percepts are noisy and give rise to overlapping perceptual distributions for signal and noise trials. In order to distinguish between signal and noise trials, the observer uses a decision criterion to classify the percepts. Signal responses are "hits" when they are correct and "false alarms" when they are incorrect; similarly, noise responses can be classified as "correct rejections" and "misses." Many criticisms of SDT have centered on how the observer places a decision criterion during a detection or classification task, and whether a deterministic criterion is used at all (see, e.g., Dorfman & Biderman, 1971; Dorfman, Saslow, & Simpson, 1975; Kac, 1969; Kubovy & Healy, 1977; Larkin, 1971).
Clearly, when initially performing a signal detection task,1 an observer may be unable to estimate stimulus distributions and payoff values accurately; thus, one might expect the placement of a decision criterion to improve with experience, approaching a static optimal criterion. Yet, some results suggest that even with extensive practice, responses can be suboptimal: There are numerous demonstrations of human probability micromatching in signal detection tasks (see, e.g., Dusoir, 1974; Lee, 1963; Thomas, 1973, 1975) and other demonstrations that static decision criteria are not typically used (e.g., Healy & Kubovy, 1981; Lee & Janke, 1964; Lee & Zentall, 1966; Treisman & Williams, 1984). Despite the fact that models accounting for these dynamics are based on a fairly reasonable assumption (i.e., that the decision criterion should improve with experience), they have not enjoyed the success of classic SDT-probably because they add layers of complexity to the theory that are not easily accommodated or validated. Given that even the basic assumptions required by SDT are rarely tested, it is perhaps not surprising that tests of these additional factors happen even less frequently.
More recently, Balakrishnan (1998a, 1998b, 1999) raised new objections to SDT on the basis of consistent violations of its assumptions: (1) Receiver operating characteristic (ROC; see below) functions produced under different base rates have different shapes (whereas SDT predicts that they should lie on top of one another), and (2) confidence-based measures typically indicate no change of the decision criterion in response to base rate manipulations (see, e. …