ROC Curves Analysis
-
Upload
ajaxtelamonio -
Category
Documents
-
view
224 -
download
0
Transcript of ROC Curves Analysis
-
7/30/2019 ROC Curves Analysis
1/20
1
ROCCurvesAnalysis
Introduction
Receiveroperatingcharacteristic(ROC)curvesareusedinmedicinetodetermineacutoffvaluefora
clinicaltest. Forexample,thecutoffvalueof4.0ng/mlwasdeterminedfortheprostatespecificantigen
(PSA)testforprostatecancer. Atestvaluebelow4.0isconsideredtobenormalandabove4.0tobe
abnormal. ClearlytherewillbepatientswithPSAvaluesbelow4.0thatareabnormal(falsenegative)and
thoseabove4.0thatarenormal(falsepositive). ThegoalofanROCcurveanalysisistodeterminethe
cutoffvalue.
Assumethattherearetwogroupsofmenandbyusingagoldstandardtechniqueonegroupisknownto
benormal(negative),nothaveprostatecancer,andtheotherisknowntohaveprostatecancer(positive).
Abloodmeasurementofprostatespecificantigenismadeinallmenandusedtotestforthedisease.
Thetestwillfindsome,butnotall,abnormalstohavethedisease. Theratiooftheabnormalsfoundby
thetesttothetotalnumberofabnormalsknowntohavethediseaseisthetruepositiverate(alsoknown
assensitivity). Thetestwillfindsome,butnotall,normalstonothavethedisease. Theratioofthe
normalsfoundbythetesttothetotalnumberofnormals(knownfromthegoldstandardtechnique)is
thetruenegativerate(alsoknownasspecificity). ThehopeisthattheROCcurveanalysisofthePSAtest
willfindacutoffvaluethatwill,insomeway,minimizethenumberoffalsepositivesandfalsenegatives.
Minimizingthefalsepositivesandfalsenegativesisthesameasmaximizingthesensitivityandspecificity.
For
the
PSA
test
abnormal
values
are
large
(>
4)
and
normal
values
are
small
(
-
7/30/2019 ROC Curves Analysis
2/20
2
1 - Specificity
0.0 0.2 0.4 0.6 0.8 1.0
Sensitivity
0.0
0.2
0.4
0.6
0.8
1.0
Albumin, A = 0.72
Figure1:AnexampleROCcurve.
AnimportantmeasureoftheaccuracyoftheclinicaltestistheareaundertheROCcurve. Ifthisareais
equalto1.0thentheROCcurveconsistsoftwostraightlines,oneverticalfrom0,0to0,1andthenext
horizontalfrom0,1to1,1. Thistestis100%accuratebecauseboththesensitivityandspecificityare1.0
sotherearenofalsepositivesandnofalsenegatives. Ontheotherhandatestthatcannotdiscriminate
betweennormalandabnormalcorrespondstoanROCcurvethatisthediagonallinefrom0,0to1,1. The
ROCareaforthislineis0.5. ROCcurveareasaretypicallybetween0.5and1.0likeshowninFigure1.
TwoormoretestscanbecomparedbystatisticallycomparingtheROCareasforeachtest. Thetestsmay
becorrelatedbecausetheyoccurredfrommultiplemeasurementsonthesameindividual. Ortheymay
beuncorrelatedbecausetheyresultedfrommeasurementsondifferentindividuals. TheROCCurves
AnalysisModulereferstothisasPairedandUnpaired,respectively,andcananalyzeeithersituation.
Thetestmeasurementsmaycontainmissingvaluesandtwomethodsareprovidedtohandlemissing
valueswhencomparingROCareaspairwisedeletionandcasewisedeletion. Thisisdescribedindetail
later.
Givenavaluefortheprobabilitythatthepatienthasthedisease(pretestprobability)theprobabilitythat
thepatienthasthedisease,giventhevalueofthetestmeasurement,canbecomputed. Also,givena
valueforthefalsepositive/falsenegativecostratio(forthescreeningexampleabove,thefalsenegative
costwould
be
greater
than
the
false
positive
cost),
an
optimal
test
value
cutoff
can
be
computed.
The
presentprogramallowsentryofthepretestprobabilityandthefalsepositive/falsenegativecostratio.
-
7/30/2019 ROC Curves Analysis
3/20
3
DataEntry
Datacan
be
entered
in
two
formats
in
SigmaPlot
Indexed
and
Grouped.
IndexedDataFormat
ThisistheformatfoundinstatisticsprogramssuchasSYSTATandSigmaStat. Indexedisthe
terminologyusedinSigmaStat. Ithasonecolumnthatindexesanothercolumn(orothercolumns). Itis
alsotheformatoftheoutputoflogisticregressionwhereROCcurvesareusedtodeterminetheabilityof
differentlogisticmodelstodiscriminatenegativefrompositivetestresults(normalsfromabnormals).
Eachdatasetconsistsofapairofcolumnsaclassificationvariableandatestvariable. Theclassification
variablehasabinarystatethatiseithernegative(normal)orpositive(abnormal).Manyprogramsusea
valueof1forpositiveand0fornegative. Theclassificationvariableisrequiredtobelocatedincolumn1
oftheworksheet. Thetestvariableisacontinuousnumericvariableandcontainsthetestresults. A
singletest
variable
will
be
located
in
column
2.
Multiple
test
variables
will
be
located
in
multiple
columns
startingincolumn2. Thereisnobuiltinlimitforthenumberoftestvariables. Thereisonlyone
classificationvariableformultipletestvariablesanditislocatedincolumn1. Thetestvariablecolumns
mustbeleftjustifiedandcontiguous. Thereforenoemptycolumnstotheleftoforwithinthedataare
allowed.
Thefollowingexampleshowsafewrowsofdatafortwodatasets. Thefirstcolumnistheclassification
variable. ItcontainsacolumntitleThyroidFunctionwhichistheclassificationvariablename. Italso
containsthetwoclassificationstatesHypothyroidandEuthyroid(normalthyroidfunction).
HypothyroidandEuthyroidaretheabnormalandnormalclassificationstates,respectively. T4andT5are
thenamesofdifferentbloodteststhatwillbeusedintheROCanalysistodiscriminatebetweennormal
andabnormalandthencomparedtodeterminewhichisthebettertest. Theclassificationvariablemust
be
in
column
1
and
the
two
test
variables
in
the
two
columns
adjacent
to
it
Theclassificationvariablenamewillbeobtainedfromthecolumn1columntitleifitexists. Thetest
nameswillbeobtainedfromthecolumntitlesofthetestvariablecolumnsiftheyexist. Theclassification
statenameswillbeobtainedfromtheentriesinthecellsofcolumn1. Ifnocolumntitleshavebeen
enteredforthetestvariablesthendefaultnamesforthetests,Test1,Test2,etc.,willbeusedand
displayedinthegraphsandreports. Thetestvariablenamesshouldbeuniquebuttheprogramwill
subscriptanyidenticalnamesthatarenot.
-
7/30/2019 ROC Curves Analysis
4/20
4
Figure2:
Indexed
data
format
for
two
tests.
The
test
names
are
T4
and
T5,
the
classificationstatesareEuthyroidandHypothyroidandtheClassificationvariable
nameisThyroidFunction. Theindexcolumnisalwayscolumn1anddata
columnsmustbeleftadjusted.
Theremustbetwoormorenonmissingdatapointsforeachtestforeachclassificationstate.Missing
valuesarehandledautomaticallybytheanalysis. Fordatacolumns,missingvaluesareeverythingbut
numericvalues(blankcells,theSigmaPlotdoubledashmissingvaluesymbol,+inf,inf, NaN,etc.).
MissingvaluesareignoredforallcomputationsexceptthePairedareacomparison(seetheMissingValue
Methodsection)wheretheyarehandledusingoneoftwopossiblealgorithms.
GroupedData
Format
Thegroupeddataformatconsistsofpairsofdatacolumnsonepairforeachtest. Onecolumninadata
pairconsistsofthenegative(normal)datavaluesandtheothercolumnforpositive(abnormal)values.
So,forexample,iftwotestsaretobecompared,theworksheetwillcontainfourcolumnsofdatathe
firsttwocolumnsforthefirsttestandthethirdandfourthcolumnforthesecondtest.
Aspecificcolumntitleformatisusedtoidentifythetestassociatedwiththedatacolumnpairandthe
classificationstateswithineachpair. Theuserisencouragedtousethisformatsinceitclearlyidentifies
thedatainthedataworksheetandwillannotateallthegraphsandreportsgenerated. Itisnotnecessary
tousecolumntitlesastheprogramwillidentifycolumnpairsstartingincolumn1withthegeneratedtest
namesTest1,Test2,etc.,andwillarbitrarilyassign1and0classificationstatenamestothefirst
andsecond
columns,
respectively,
but
this
is
clearly
not
the
best
way
to
organize
the
data.
Since
the
test
namesandclassificationstatesarenumericalitisalsomoredifficulttointerprettheresults.
ColumnTitleConventionforGroupedData
ThiscolumntitleconventionisasimplewaytoidentifyworksheetdatafortheGroupeddataformat. The
followingexampleshowsafewrowsfortwodatasets. ThefirsttwocolumnscontainthedatafortheT4
-
7/30/2019 ROC Curves Analysis
5/20
5
test. ThefirstcolumnT4 EuthyroidisthecolumnwiththenormaldatafortestT4. Thecolumntitle
consistsofthetestnamefollowedbyaminussignfollowedbytheclassificationstate. Spacesoneither
sideoftheminussignareignored. ThesecondcolumnT4 Hypothyroidisthecolumnwiththe
abnormaldatafortestT4. Thethirdandfourthcolumntitlesarethesameasthefirsttwoexceptthe
secondtestnameT5isused.
Figure3. Groupeddataformatfortwotests. ThisisthesamedataasinFigure
1. TherearetwotestsT4andT5. Eachtestconsistsofapairofdatacolumns. In
thiscaseT4isincolumns1and2andT5incolumns3and4. TheTestState
columntitleformatisusedtoidentifythetwotestsandthenormal(Euthyroid)
andabnormal(Hypothyroid)states.
Thetestnamesinbothcolumnsofacolumnpairmustbethesame. Alsotheremustbeexactlytwo
classificationstatesinthecolumntitles.
Likethe
Indexed
format,
missing
values
in
the
worksheet
cells
are
ignored
except
for
special
handling
whencomparingROCareas(seetheMissingValueMethodsection).
ProgramOptions
SelectingROCCurvesfromtheSigmaPlotToolboxmenuopensthedialog
-
7/30/2019 ROC Curves Analysis
6/20
6
TestandclassificationstatenamesfromtheindexeddatashowninFigure2oftheDataEntrysectionare
displayedinthisdialog.
DataSelectionOptions
DataFormat(AutomaticDetermination)
Inmostcasetheprogramwillidentifythedataformatfromtheinformationinthedataworksheet. Inthe
dialogabovetheformatwasidentifiedasIndexed. Youmayselectfromthetwoformats Indexedand
Grouped.
AvailableDataSetsSelectedDataSets
SelectoneormoreoftheavailabledatasetsbyclickingonthemintheAvailableDataSetswindowand
thenclickingontheAddbutton. Ifdesired,youmaythenselectatestnameintheSelectedDataSets
windowandclickRemovetodeselectthetest.
-
7/30/2019 ROC Curves Analysis
7/20
7
DataType
IftwoormoredatasetsareselectedthentheDataTypeoptionforcorrelatedtestsismadeavailable
YoumayselecteitherPaired,forcorrelatedtests,orUnpaired. IfPairedisselectedtheROCareasand
areacomparisonsaredeterminedusingtheDeLong,DelongandClarkePearsonmethod(2). IfUnpairedis
selectedtheareasarecomputedusingtheHanleyandMcNeilmethod(3)
andtheareasarecompared
usingaZtest.
MissingValueMethod
IfmissingvaluesexistthentwooptionsareavailableforthepairwisecomparisonofROCareasPairwise
DeletionandCasewiseDeletion. Thisoptionisnotavailableifnomissingvaluesexist.
Pairwisedeletiononlydeletesrowscontainingmissingvaluesfortheparticularpairbeinganalyzednot
foranentirerowofdata. Fewerdatavaluesaredeletedusingthismethod. Therearesituationswhen
pairwisedeletionwillfailbutthisistheoptiontousewhenitispossible. Casewisedeletiondeletesall
cellsinanyrowofdatacontainingamissingvalue.Muchmoredatamaybedeletedusingthisoption. To
betterunderstandthedifference,considerasimpleexampleoftwodatacolumnsofequallengthoneof
whichhasnomissingvaluesandtheotherhasonemissingvalue.WhenROCareasarebeingcompared,
certaincomputationsonthesetwocolumnswillbedonepairwisethefirstcolumnwithitself,thefirst
columnwiththesecondcolumnandthesecondcolumnwithitself.Whenthecolumnwithoutamissingvalueisbeingcomparedwithitselfnorowdeletionsoccurforpairwisedeletion. Forcasewisedeletion,
however,therowthatcontainsthemissingvaluewillbedeletedfrombothdatasets. So,forcasewise
deletion,thecomputationinvolvingthecolumnwithoutamissingvaluewithitselfwillbedonewithone
rowdeleted(therowcorrespondingtothemissingvalueintheotherdataset). Theprogramdetermines
whenpairwisedeletionisnotvalidandinformstheuserwhenthisisthecase.
-
7/30/2019 ROC Curves Analysis
8/20
8
PositiveStateOptions ClassificationStateandDirection
ThetwoclassificationstatesarereferredtoasNegative(normal)orPositive(abnormal). TheROC
analysissoftwaremustbeinformedwhichstateisPositiveandwhetherthetestmeasurementvalues
forthepositivestateareHigh,meaninghigherthanthoseofthenegativestate,orLow,meaning
lowerthanthoseofthenegativestate.
AcceptednormalvaluesforthePSA(prostatespecificantigen)testarelessthan4ng/mlandabnormal
valuesarehigherthanthis. Thusifthetwoclassificationstatesnamesarepositiveandnegativethen
thePositivestateispositiveandthePositiveDirectionisHigh. Inthiscaseyouwouldselecttheradio
buttonnexttopositiveandHigh.
Ontheotherhand,fortheT4(thyroxine)testforhypothyroidismtheT4valuesarelowerintheabnormal
statethanforthenormalstate. InthiscasetheabnormalPositiveStateisHypothyroidandthePositive
DirectionisLow. SoyouwouldselecttheradiobuttonnexttoHypothyroidandLow.
Whathappensifyouselecttheincorrectoption? Sensitivity(specificity)isdefinedintermsofthepositive
(negative)state. Soifthepositivestateisincorrectlyselectedthensensitivityandspecificitywillbe
incorrectlydefined(switched)andtheROCcurvewillhavetheXandYaxesswitched. Thiswillresultinan
ROCcurvethatappearsbelowthediagonalunityline. Itwillhaveanarealessthan0.5. Theprogramwill
detectthisandgiveyoutheoptions
-
7/30/2019 ROC Curves Analysis
9/20
9
ItispossiblethatthereissomethingwrongwiththedatasoyoucanAborttheanalysisandcorrectthe
problem.MorelikelyyouhaveselectedtheincorrectpositivestateordirectionsoyoucanRetrythe
analysiswithcorrectselections. Inrareoccasionsformultipletestssometestswillhaveareasgreater
than0.5andoneormorewillhaveareaslessthan0.5. InthiscaseyoucanIgnorethiswarningand
continuewiththeanalysis.
ReportOptions
ConfidenceIntervals
ConfidenceintervalsarecomputedforstatisticsinboththeSensitivity&SpecificityandAreaComparison
reports. Youcangenerate90,95and99%confidenceintervals.
CreateSensitivityandSpecificityReport
Cutoffvaluesarecreatedbetweeneachtestdatavalueinthe(sorted)dataset. Iftherearealarge
numberof
data
points
and
several
tests
then
there
will
be
alarge
number
of
cutoff
values
and
the
Sensitivity&SpecificityReportcanbeverylong. Thecheckbox
allowsyoutoturnoffthisreport. Ifyouturnoffthisreportthenallreportoptionsinthedialogbelowthis
arenotrequiredandaredisabled.
Fractions/Percents
Youmaydisplaysensitivities,specificitiesandprobabilitiesineitherfractionorpercentformat. Selecting
Percentsalso
requires
the
pre
test
probability
to
be
entered
as
apercent.
CreatePostTestResults
Selectingthisoptionallowsentryofthepretestprobability. Italsoenablesthepossibleentryofthe
falsepositive/falsenegativecostratio. Givenapretestprobabilitytheprogramwillcreateposttest
probabilities,boththepositivepredictivevalue(PV+=probabilityofdiseasegivenapositivetestresult)
andthenegativepredictivevalue(PV =probabilityofnodiseasegivenanegativetestresult),foreach
cutoffvalue. Ifthecostratiooptionisselectedthentheoptimalcutoffvaluewillbecomputed. Allof
theseresultsaredisplayedforeachtestintheSensitivity&Specificityreport.
ROCGraph
Options
AllofthegraphoptionsinthedialogapplytotheROCgraph. Theyallowyoutoaddadiagonallinetothe
graph,addgridlines,addsymbolsforsensitivityandspecificityateachcutoffpointandchangetheROC
plotlinesfromsolidtodifferentlinestyles.
AnalysisResults
-
7/30/2019 ROC Curves Analysis
10/20
10
Introduction
TypicalresultsoftheROCanalysisareshowninthefollowingexamplefromtheNotebookManager.
ThefirstsectionentitledOvarianCancercontainstheworksheetcontainingtherawdata. Theprogram
createdthenextthreesectionsthatcontaintwographsandtworeports. Thecontentsofthetwographs
ROCCurves
DotHistogram
andthetworeports
Sensitivity&Specificity
ROCAreas
aredescribedinthenextsections.
ROCCurvesGraph
TheROCcurvesgraphforthreedatasetsisshowninFigure4. Thesegraphsarederivedfromnumerical
resultsintheworksheetentitledGraphData. Thegraphtitleisobtainedfromthesectionname
containingtherawdata. ThelegendshowsthetestnamesandtheROCareasforeachcurve. The
diagonallineandgridsoptionswereselectedforthisgraph.
-
7/30/2019 ROC Curves Analysis
11/20
11
Ovarian Cancer ROC Curves
1 - Specificity
0.0 0.2 0.4 0.6 0.8 1.0
Sensitivity
0.0
0.2
0.4
0.6
0.8
1.0
US, A = 0.85
CT, A = 0.93
MR, A = 0.99
Figure4. TheROCcurvesgraphforthreetests.
Ofcoursethisgraphcanbeeditedinanywayyouwish. Youmightwanttochangethestartingcolorof
thecolorschemeusedforthelinecolors. YoucandothisbydoubleclickingononeoftheROCplotlines
andthenrightclickingontheLineColorlistboxasshownnext.
DotHistogramGraph
DothistogramsforthedataassociatedwiththeROCcurvesinFigure4areshowninFigure5below.
-
7/30/2019 ROC Curves Analysis
12/20
12
Ovarian Cancer Data
US-normal
US-abnormal
CT-norm
al
CT-abnormal
MR-norm
al
MR-abno
rmal
TestValue
0
2
4
6
8
10
12
14
Cutoff < 7.13
Sens = 0.78
Spec = 0.85
Cutoff < 7.14
Sens = 0.84
Spec = 0.97
Cutoff < 8.34
Sens = 0.94
Spec = 0.97
Figure5: Dothistogrampairsforeachtest. Thehorizontallinesandthetables
belowthegraphshowtheoptimalcutoffvaluesdeterminedfromthepretest
probabilityandcostratio.
Thegraphtitleisobtainedfromthetitleofthesectioncontainingtherawdata. Thexaxisticklabelsare
obtainedfromthetestnamesandtheclassificationstatenames. Theticklabelswillrotateiftheyaretoo
longto
fit
horizontally.
The
symbol
layout
design
allows
for
symbols
to
touch
horizontally
and
nest
vertically.
Ifvaluesforpretestprobabilityandfalsepositive/falsenegativecostratioareenteredthentheoptimal
cutoffvaluesforeachtestarecomputedandrepresentedasahorizontallineacrossthetwodot
histogramsforeachtest. Thenumericvaluesfortheoptimalcutoffparametersareshownastables
belowthexaxis.
Sensitivity&SpecificityReport
Thesensitivity&Specificityreportcontainsresultsforalltestswithadditionaltestsresultsplacedin
reportrows
below
those
of
prior
tests.
The
results
for
each
test
can
be
separated
into
three
parts:
1)
optimalcutoffvalue,2)sensitivityandspecificityversuscutoffvaluesand3)likelihoodratiosandpost
testprobabilities.
Ifvaluesforbothpretestprobabilityandcostratiohavebeenenteredthentheoptimalcutoffis
calculated. AslopeofthetangenttotheROCcurvemisdefinedintermsofthetwoenteredvalues(P=
pretestprobability)(1)
-
7/30/2019 ROC Curves Analysis
13/20
13
1false positive c os t Pm
false negative cos t P
=
(1)
Theoptimalcutoffvalueiscomputedfromsensitivityandspecificityusingtheslopembyfindingthe
cutoffthatmaximizesthefunction(1)
( )1Sensitivity m Specificity (2)
TheresultsofthiscomputationintheSensitivity&SpecificityreportareshowninTable1.
Table1: OptimalcutoffresultsintheSensitivity&Specificityreport.
Forthisdataset,theoptimalcutoffis7.125forapretestprobabilityof0.5andcostratioof1.0.
Sensitivities,specificitiesandtheirconfidenceintervalsarelistedasafunctionofcutoffvalueinthe
secondpartofthereport. AportionoftheseresultsisshowninTable2. Theseresultscanbeexpressed
asfractionsorpercentsbyusingtheFractions/Percentsoption.
-
7/30/2019 ROC Curves Analysis
14/20
14
Table2: SensitivityandspecificityresultsintheSensitivity&Specificityreport.
ThethirdpartoftheSensitivity&Specificityreportcontainsthelikelihoodratiosandposttest
probabilities.
Thepositiveandnegativelikelihoodratiosaredefinedrespectivelyas
1
Pr obability of a positive test given the presenceof disease SensitivityLR
Pr obability of a positivetest given the absenceof disease Specificity+ = =
(3)
1Pr obability of a negativetest given the presence of disease SensitivityLR
Pr obability of a negative test given the absenceof disease Specificity
= = (4)
Theposttestprobabilitiesaretheprobabilityofdiseasegivenapositivetest(PV+)andtheprobabilityof
nodiseasegivenanegativetest(PV). Thesewillbecomputedwhenapretestprobabilityhasbeen
entered.Using
P=pre
test
probability,
the
equations
used
for
these
probabilities
are
( ) ( )1 1Sensitivity x P
PVSensitivity x P Specificity x P
+ =+
(5)
( )
( ) ( )
1
1 1
Specificity x PPV
Specificity x P Sensitivity x P
=
+ (6)
AportionofthereportshowingthelikelihoodandposttestprobabilitiesresultsisshowninTable3.
-
7/30/2019 ROC Curves Analysis
15/20
15
Table3: Positiveandnegativelikelihoodratios,LR+andLR,andposttest
probabilities,PV+andPV,intheSensitivity&Specificityreport.
Thepositivelikelihoodratioisnotdefinedforsomecutoffvaluessincespecificity=1.
ROCAreas
Report
TheROCAreareportconsistsoftwoparts:1)ROCareasandtheirassociatedstatisticsand2)pairwise
comparisonofROCareas. AnexampleofareportisshowninTable4.
-
7/30/2019 ROC Curves Analysis
16/20
16
Table4: AnexampleROCAreasreport. Fromtoptobottomitshowsthetypeof
analysisusedtogetherwiththemissingvaluemethod,theROCareasand
associatedstatisticsandapairwisecomparisonofROCareas.
Inthis
case
there
are
three
correlated
tests.
Row
two
of
the
report
shows
that
aPaired
Analysis
was
performedand,sincethereweremissingvaluesinthedata,PairwiseDeletionofmissingvalueswas
selectedtocomparetheareas.
ThefirstsectionofthereportshowstheROCcurveareasforthethreetests. Thisisfollowedbythe
standarderroroftheareaestimate,the95%confidenceinterval(90%and99%arealsoavailable)andthe
Pvaluethatdeterminesiftheareavalueissignificantlydifferentfrom0.5. Thesamplesizeandthe
numberofmissingvaluesforeachclassificationstatearegiven. Thenumberofmissingvaluesreflects
onlywhatisseeninthedataanddoesnotgivethenumberusedforeachcomputationpairinthe
pairwisedeletedcomparisonofareas.
Thesecondsectionshowstheresultsofthepairwisecomparisonofareas. ThemethodofDeLong,
DeLongandClarkePearson(2)
isusedtocompareareaswhenthePaireddatatypeoptionisselected.
Whenthe
Unpaired
data
type
is
selected,
areas
are
compared
using
aZtest.
The
report
shows
results
for
allpairsofdatasets. Thedifferenceofeachareapairanditsstandarderrorand95%confidenceinterval
arecomputed. Thisisfollowedbythechisquarestatisticfortheareacomparison(orZstatisticif
Unpairedisselected)anditsassociatedPvalue.
-
7/30/2019 ROC Curves Analysis
17/20
17
FormattedFullPrecisionDisplay
Thisreportpresentsthenumericresultsinafoursignificantdigitformatwithfullprecisionavailable.
Doubleclickonanycell(excepttheconfidenceintervals)todisplaythenumberatfullprecision.
AdditionalGraphs
Resultsdatainbothreportscanbeusedtocreateadditionalgraphs. Someexamplesseeninthe
literatureareshownhere.
SensitivityandSpecificityvs.Cutoff
ThedataforthegraphinFigure6isfromtheSensitivity&Specificityreportincolumns1,2and4. Use
theDataSamplingoptioninGraphProperties,Plots,Datatospecifytherowrangeforthegraph(youcan
alsodragselecttherowsintheworksheettodothis).
Cutoff
0 2 4 6 8 10 12 14
Sensitivity
0.0
0.2
0.4
0.6
0.8
1.0
S
pecificity
0.0
0.2
0.4
0.6
0.8
1.0
Figure6:Graphofsensitivityandspecificityvs.cutoffforonetestusingdata
fromcolumns1,2and4oftheSensitivity&Specificityreport.
LikelihoodRatios
ThepositiveandnegativelikelihoodratiosforthreedifferentimagingmodalitiesareshowninFigure7
(thedataisartificial). Thedataisincolumns1,6and7oftheSensitivity&Specificityreport. Thevalues
associatedwiththeoptimalcutoffareshownassolidsymbols. Thelargestpositivelikelihoodand
smallestnegativelikelihoodattheoptimalcutoffisassociatedwithmagneticresonanceimaging(MR).
-
7/30/2019 ROC Curves Analysis
18/20
18
Likelihood Ratio of a Positive Test
Cutoff
4 5 6 7 8 9 10
LR
+
0
20
40
60
80
US
CT
MR
Likelihood Ratio of a Negative Test
Cutoff
4 5 6 7 8 9 10
LR-
0.0
0.2
0.4
0.6
0.8
1.0
1.2
US
CT
MR
Figure7: Positiveandnegativelikelihoodratiosgraphedfromdatainthe
Sensitivity&Specificityreportfromcolumns1,6and7. Theresultsforthree
testsareshowntogetherwithvaluesassociatedwiththeoptimalcutoff(solid
symbols).
OptimalCutoffvs.CostRatio
Frequentlyitcanbedifficulttodetermineavalueforthefalsepositive/falsenegativecostratio. Soitis
worthperformingasensitivityanalysis(sensitivityheremeanshowmuchonevariablechangeswith
changesinasecondvariable)toseewhetherthecutoffvaluechangessignificantlyintherangeofcost
ratiovaluesofinterest. TheROCCurvesModulewasrunmultipletimesfordifferentcostratiosanda
graphofoptimalcutoffvs.costratioforthethreeimagingmodalitytestsisshownbelow.
Cost Ratio
0.1 1 10
OptimalCutoff
4
6
8
10
12
14
US
CTMR
Figure8: Optimalcutoffvaluesobtainedfrommultiplerunsoftheprogram.
Regionsofinsensitivity,orstrongsensitivity,tocostratiocanbeidentified.
Iftherelativecostofafalsepositiveismuchgreaterthanthatofafalsenegativethenthecostratiois
greaterthan1. Butletsassumethatwedontknowexactlyhowmuchgreateritisbuthavesomeidea
thatitshouldbeintherangeof2to5,say. Lookingattheoptimalcutoffforthebestimagingmodality
-
7/30/2019 ROC Curves Analysis
19/20
19
(MR,greenline)wefindthatitdoesntchangeforcostratiosfrom2to20. Sotheoptimalcutoffis
insensitivetocostratioand,inthiscase,itisnotimportanttoknowaprecisevalueforcostratio.
PostTestProbabilityvs.PreTestProbability
Givenvaluesofsensitivityandspecificityassociatedwiththeoptimalcutoffagraphofposttest
probabilitiesasafunctionofpretestprobabilitycanbecreatedusingequations(5)and(6). Theposttest
probabilityofdiseasewhenthetestispositive,bluelinesinFigure9,wasobtainedfromequation(5)and
theposttestprobabilityofdiseasewhenthetestwasnegative,redlines,wasobtainedfrom1.0minus
equation(6). AtransformwaswritteninSigmaPlotimplementingthesetwoequationsthatgeneratedthe
posttestprobabilitiesforarangeofpretestprobabilities. Theresultsforthebesttest,MR,andworst
test,US,areshown. TheMRtestisclearlybettersincetheposttestprobabilityrange,fromnegativetest
topositivetest,islarger. Thusgivenapositivetestthepatientismorelikelytohavethediseaseusingthe
MRtestratherthantheUStest. Similarly,givenanegativetestitislesslikelythatthepatienthasthe
diseaseusingtheMRtest.
Imaging Modalities
Pre-Test Probability
0.0 0.2 0.4 0.6 0.8 1.0
Post-TestProbability
0.0
0.2
0.4
0.6
0.8
1.0
US
MR
Test Positive
Test Negative
Figure9: Posttestprobabilitiesofdiseasegivenpositiveandnegativetest
results. TheMRtestisbasedonsensitivity=0.94andspecificity=0.97whereas
theUStestusedsensitivity=0.78andspecificity=0.85.
-
7/30/2019 ROC Curves Analysis
20/20
20
References
1. Zweig,MH,Campbell,G.Receiveroperatingcharacteristic(ROC)plots:Afundamentalevaluation
toolinclinicalmedicine. ClinChem1993;39/4,56577.
2. DeLong,ER,DeLong,DM,ClarkePearson,DL. Comparingtheareasundertwoormore
correlated
receiver
operating
characteristic
curves:
a
nonparametric
approach.
Biometrics
1988;44,837845.
3. Hanley,JA,McNeil,BJ. Themeaninganduseoftheareaunderareceiveroperating
characteristic(ROC)curve. Radiology1982,143,2936.