Medical Imaging Techniques: An Illustrative Study
Many new diagnostic tests and many new, expensive imaging modalities are introduced into the health care system each year. Evaluating them can be difficult, particularly if alternative means exist to approximately the same diagnostic end. Yet, lest new techniques be introduced haphazardly, critical protocols and methods must be available so that they can be promptly assessed. We are currently completing for the National Cancer Institute a general protocol for the evaluation of diagnostic devices, with an emphasis on imaging modalities. The present study was undertaken both to refine and to illustrate this protocol ( 1).
To date, most comparative studies of imaging systems have taken one or the other of two inadequate approaches. One has been to measure fidelity--how well the system reveals the presence and detail of a standard test object, called a "phantom." The drawback of this approach is that what is measured is only the potential for mediating accurate detection and diagnosis of real lesions. The other approach, although based on using real cases and examining actual diagnostic performance, has been inadequate because of the simplistic accuracy indices that are usually obtained. Indices such as the proportion of true-positive responses, single true-positive false-positive pairs (and their ratio), single pairs of "sensitivity" and "specificity" values (and their sum), and agreement scores do not control for the influence of the reader's confidence threshold or "decision criterion," that is, the tendency to overcall or undercall disease, nor for the prevalence of disease in the study population at hand.
These two inadequacies are overcome in the approach of our general protocol. It specifies a procedure in which real cases and real diagnostic tasks are used and in which, moreover, performance is scored in relation to independent, external evidence. It further specifies a psychophysical method that generates perfor-