1

ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE

RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

JOSEPH GEORGE CALDWELL, PH.D.

27 FEBRUARY 2007

2

BACKGROUND

• TWO-ALTERNATIVE STATISTICAL DECISION PROBLEMS: HOW TO ASSESS ACCURACY?

• FROM A SERIES OF TEST RESULTS (E.G., BLOOD TESTS, X-RAYS, MRI IMAGES), MUST DECIDE WHETHER SOMEONE HAS A PARTICULAR DISEASE OR CONDITION

• STATISTICAL CONSULTING REPORT: “DESCRIPTION OF THE STATISTICAL SUBSYSTEM OF THE AUTOMATED RECEIVER OPERATING CHARACTERISTIC (ROC) SYSTEM”

3

THE TWO-ALTERNATIVE STATISTICAL DECISION PROBLEM

THE STIMULUS-RESPONSE MATRIX:

TYPE 2 ERROR

TYPE 1 ERROR

4

HOW TO REPRESENT THE ACCURACY OF THE DECISION PROBLEM?

• BY VARYING THE DECISION CRITERION, THE PROBABILITIES OF THE TWO TYPES OF ERROR MAY BE ADJUSTED -- A TRADE-OFF

• BY CHANGING THE DIAGNOSTIC PROCEDURES (E.G., ADDING TESTS, IMPROVING TESTS), THE PROBABILITIES OF BOTH TYPES OF ERRORS MAY BE REDUCED

• A PROBLEM: TOO MANY STIMULUS-RESPONSE MATRICES. HOW TO SUMMARIZE ACCURACY MORE SUCCINCTLY?

5

THREE EXAMPLES

6

THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

7

AN EXAMPLE

8

AN EXAMPLE (CONT.)

9

A TECHNICAL COMMENT…

THE SLOPE OF THE ROC CURVE AT ANY POINT IS EQUAL TO THE LIKELIHOOD-RATIO CRITERION THAT GENERATES THE POINT.

THE LIKELIHOOD FUNCTION IS: )|(

)|()(

nef

sefeL .

BY DEFINITION,

k

desefsSP )|()|( AND

k

denefnSP )|()|( .

THE SLOPE OF THE ROC CURVE AT THE POINT DETERMINED BY THE CRITERION k IS:

)()|(

)|(

)|(

)|(kL

nkf

skf

nSdP

sSdP

k

10

EXAMPLES OF ROC CURVES

11

APPLICATION TO LOGISTIC REGRESSION

j

ijjiiii xppp xβ'))1/(log()(logit

ESTIMATE

))exp(1/(1 xβ estestestp

DECISION CRITERION: DECIDE “YES” IF pest > c AND “NO” IF pest <= c

VARY c AND CALCULATE THE PROPORTION OF HITS AND FALSE POSITIVES, AND PLOT ON A ROC GRAPH (EACH VALUE OF c YIELDS A DIFFERENT POINT ON THE CURVE).

LOGISTIC REGRESSION MODEL:

12

ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

13

ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

14

REFERENCES

15

REFERENCES

16

REFERENCES

1. Swets, John A., and Ronald M. Pickett, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory, Academic Press, New York, 1982

2. Green, David M. and John. A Swets, Signal Detection Theory and Psychophysics, Wiley, 1966, reprinted by Peninsula Publishers, Los Altos, CA, 1988

3. Dorfman, Donald D. and Edward Alf, Jr., "Maximum-Likelihood Estimation of Parameters of Signal-Detection Theory and Determination of Confidence Intervals -- Rating Method Data", Journal of Mathematical Psychology, vol. 6, 1969, pp. 487-496

4. Ogilvie, John C. and C. Douglas Creelman, "Maximum-Likelihood Estimation of Receiver Operating Characteristic Curve Parameters," Journal of Mathematical Psychology, vol. 5, 1968, pp. 377-391.

5. Rau, C. R., Linear Statistical Inference and Its Applications, Wiley, New York, 1965, pp. 302-3096. Ralston, A., A First Course in Numerical Analysis, McGraw Hill, New York, 1965, p. 3487. Caldwell, J. G., Description of the Statistical Subsystem of the Automated Receiver Operating

Characteristic System, 19958. Logistic Regression Examples Using the SAS System, SAS Institute, Cary, NC, Version 6, First

Editioin, 19959. Bamber, D. (1975), “The Area Above the Ordinal Dominance Graph and the Area Below the

Receiver Operating Characteristic Graph,” Journal of Mathematical Psychology, 12, 387-415.10. Hanley, J. A. and McNeil, B. J. (1982), “The Meaning and Use of the Area Under a Receiver

Operating Characteristic (ROC) Curve,” Radiology, 143, 29-36.