Signal Detection Theory and Roc Analysis in Psychology and Diagnostics: Collected Papers

ing to improve on, a given technique or system for diagnosis ( Swets & Pickett, 1982).

The bonus that the appropriate detection-theory model carries along is the ability it provides, via the ROC, to obtain a relatively pure index of discrimination capacity--one largely independent of the decision criterion or choice tendency--and also an index of the decision criterion that is operative in any given instance. Experimental psychology and practical fields thereby gain a valid and reliable index of discrimination capacity. Psychology, especially, acquires an ability to determine whether various variables that effect a change in performance do so by affecting discrimination acuity or the decision criterion (Chapter 1). An example here is the finding that the declining hit rate observed in perceptual vigilance experiments is often the result of an increasingly strict criterion rather than of decreasing sensitivity ( Parasuraman, 1984).

Practical fields need a criterion-free index of discrimination capacity when the criterion used with a given system varies widely over the different settings in which that system is used, and for which it is being evaluated. Thus, for example, the strictness of the criterion used with a particular imaging system in clinical medicine can be quite different in screening and referral settings, and the criterion used with a weather forecasting system will differ from one geographical region to another and from one user of forecast information to another.

Practical fields, moreover, acquire an ability from the ROC to assess the efficacy of a diagnostic system for a specific setting. In a given setting, one is fundamentally concerned with some measure of the system's utility, for example, its expected value or payoff, as determined by the probabilities of the various outcomes of the decision and by the benefits and costs of those outcomes. For specific settings in which the probabilities, benefits, and costs are stable and can be estimated, the emphasis is more on the payoff associated with a particular point on the ROC--that is, with a particular decision criterion--than on an index of the locus of all ROC points. For any of several decision rules that seek to maximize one or another quantity related to utility, one can calculate the optimal decision criterion, or operating point on the ROC ( Green & Swets, 1966/ 1988; Swets & Pickett, 1982; Swets & Swets, 1979). And then, usually, the system can be adjusted to operate at or near that criterion or point. Because the binormal slopes of empirical ROCs vary widely from one instance to another, in a manner so far not predictable, both the calculation of, and adjustment to, the optimal criterion depend on having the empirical ROC in hand.

Bacus, J. W. ( 1982). Application of digital image processing techniques to cytology automation (Tech. Rep.) Chicago: Rush Presbyterian-St. Luke's Medical Center, Medical Automation Research Unit.

-56-