Academic journal article Journal of Evidence-Based Psychotherapies

Definition of Efficiency Indices for Three-Category Diagnostic Tests

Academic journal article Journal of Evidence-Based Psychotherapies

Definition of Efficiency Indices for Three-Category Diagnostic Tests

Article excerpt

(ProQuest: ... denotes formulae omitted.)


In psychological measurement it is usual to determine cut-off scores for a test according to a criterion variable (e.g. diagnostic criterion) in order to classify subjects into mutually exclusive categories.

Classification problems with diagnostic tests involve a predictor variable (test, questionnaire, etc.) and a criterion variable (diagnostic test). If measures are obtained on ordered scales, high test scores should correspond to subjects with higher probability of a disorder. If the diagnostic test provides two or more categories to which a subject may, in fact, belong, one or more cut-off scores can be established in the predictor variable (test). These cut-off scores classify the subject's scores in the test into two or more categories.

A 2 x 2 classification is obtained when a cut-off score is established on test scores and the diagnostic test establishes two categories. Efficiency indices frequently used (sensitivity, specificity and their associated errors) are associated with the validity of a 2 x 2 classification.

If the diagnostic test establishes three categories, and two cut-off scores are established in the test scores, a measure of goodness of classification (efficiency or usefulness) must be given. This measure quantifies the relationship between the classification established by these two cut-off scores in the test and the criterion or diagnostic test.

Several authors have proposed measures for the usefulness of the classifications, considering three or more categories, provided by ROC analysis. Hand and Till (2001) present a generalization of the AUC measure for classifiers that assign a different score or probability to each prediction, defining a measure, M, from the AUC associated with each pair of categories. Ferri, Hernández- Orallo, and Salido (2003) give a set of efficiency indices (AUC extension for a point, AUC-1PT3; and two variants of the Hand and Till measure, HT1 and HT3). Sampat et al. (2009) provide a critical review of a large number of efficiency measures, highlighting their advantages and drawbacks.

This paper defines one efficiency index to evaluate the usefulness of two cut-off scores (set in test scores) based on three categories set out in a criterion variable (diagnostic test): the Index based on the Projected Surface (IPS). Based on the calculation procedure given by Rivas and Caballero (2011), it is also possible to obtain the Index based on the Tetrahedron Volume (ITV) by using the IPS.

An example of application is proposed in this paper in order to illustrate the procedure to calculate the IPS and the IVT. From the three categories established by the criterion variable and the two cut-off scores (obtained on test scores with different procedures such as logistic regression and ROC analysis), IPS and ITV values are given. These values are compared with those of other efficiency indices (AUC-1PT3, HT1 and HT3) defined by Ferri et al. (2003).

The Area Under the ROC Curve (AUC) is the traditional measure employed in a ROC analysis with only two categories. This index is the most suitable when the types of error, False-Negative and False-Positive proportions, are equally meaningful. If, for any reason, one of these errors is more significant than the other, it would be convenient to take into account the cost involved in committing such an error. The possibility of including weights in the index definition may be considered. Before defining IPS and ITV indices it is necessary to extend to a three-category classification the AUC index for two-categories. The AUC is described in Appendix A, where is shown to be the same that the Area Above the ROC Curve (AAC) given in Ferri et al. (2003). In Appendix B, this measure is extended for a classifier in a three-category classification.

Index based on the Projected Surface

The Index based on the Projected Surface (IPS) is a goodness-of- classification index for three categories, based on the AUC index for two categories. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.