The Receiver-Operating Characteristic (Roc) Analysis: Fundamentals and Applications in Clinical Psychology

Article excerpt

Abstract

The Receiver-Operating Characteristic (ROC) analysis has been long used in Signal Detection Theory to depict the tradeoff between hit rates and false alarm rates of classifiers. In the last years, ROC analysis has become largely used in the medical community for visualizing and analyzing the performance of diagnostic tests. Our article points out some fundamental aspects of ROC analysis underlying the importance of using ROC analysis in evaluating the diagnostic validity of tests commonly used in clinical psychology. The main statistical programs available for this type of analysis, with their advantages and deficiencies are also discussed. In order to illustrate how ROC analysis works in clinical research, we also describe an application of ROC analysis in evaluating scales generally related to depression.

Keywords: receiver operating characteristics (ROC), ROC analysis, area under the curve (AUC), diagnostic performance, sensitivity, specificity, clinical psychology, depression

THE ROC ANALYSIS: FUNDAMENTALS

Receiver Operating Characteristic (ROC) analysis is a procedure used in assessing diagnostic properties of tests, namely in assessing the way various measures generally discriminate between different categories of subjects. In order to do this, a cut-off point needs to be established; based on the cut-off point, we can determine whether a person with a certain score belongs to one category or another (e.g. normal/non-clinical or clinical group). ROC analysis may also be used when comparing the diagnostic performance of two or more tests (Westin, 2001).

ROC analysis was used for the first time in the military field, for the analysis of radar images, during The Second World War (Westin, 2001). In medicine, the procedure has been used since the 1960s, and there is an extensive literature on the use of ROC graphs for diagnostic testing (Fawcett, 2006). In chemistry, ROC curves analysis is used to solve dichotomous decision problems such as: the presence or absence of a protein marker, whether the structure of a molecule is X or Y, whether a reaction obeys first order kinetics or second, should a reaction be terminated or continued, etc.? (Brown & Davis, 2006).

In clinical psychology, ROC analysis is being used with increased frequency, particularly in examining the utility or performance of diagnostic or screening tools for: future difficulties in reading comprehension (Shapiro, Solari & Petscher, 2008), alcohol and drug abuse (Kills Small et al., 2007), neuropsychological impairment (Horwitz et al., 2008; O'Brien et al., 2007), depression (Benazzi, 2008; Serrano-Duenas & Serrano, 2008; Stafford, Berk & Jackson, 2007; Ballesteros et al., 2007; Walsh et al., 2006), obsessive-compulsive disorder (Ivarsson & Larsson, 2008), bipolar disorder (Parker et al., 2008;), suicide (Jokinen, Nordstrom & Nordstrom, 2008), dementia (Chiu et al., 2008; Giaquinto & Parnetti, 2006), dropout risk from different treatments such as cognitive-behavior therapy for insomnia (Ong, Kuo & Manber, 2008) and so on.

Two of the more important literature reviews on ROC curves analysis have been conducted by McFall & Treat (1999) and Streiner & Cairney (2007). McFall and Treat follow a broader objective in their work, as they themselves state, "about the functions of clinical assessment, the standards by which methods can be evaluated, and the most promising approaches to achieving the broad goals of clinical assessment" (p. 216). Compared to their article and also to the one of Streiner and Cairney, our paper is intended to be more applicative. We use actual data collected from a Romanian sample to illustrate a number of ROC concepts reviewed in this paper; we also discuss an extended set of uses, from establishing cutoff-points to comparing different tests to overall and partial ROC areas, to specific points of the curves.

Measures related to ROC curves

The dichotomous decision process is based on a threshold value ? …