Reducingbiasinforensic&clinicaldecision-makingJOS EP H LOCK H A RT, P H D, A BP P

S ATY S ATYA - MURTI, MD, FA A N

1.Lockhart:◦ Introduction◦ Definition andethical issues◦ Major Forensic Examples (asperDror)

2.Satya-Murti:◦ Ubiquity ofbias◦ Types ofBias

◦ Day-to-day,andclinicalexamples

3.Lockhart:Allegiance bias and“blind spot” bias

4.Satya-Murti:De-biasing research in clinical arenaOutline ofDe-Biasing techniques

5.Lockhart:Biasprevention ideasBiasand theexpertwitness

OOuuttlliinneeooffTThhiissPPrreesseennttaattiioonn

0:00 ~0:60

Q&A

Shouldweworryaboutbiasindecision-making?

Biasispervasive.Itaffectsevery,andall,aspectsoflife

DAYTODAY LIFE

HistoryWarfarePolitics

CLINICALMEDICINE

FORENSIC PRACTICE

Anexamplefromledzeppelin

AworkingdefinitionofconfirmationBias

“In[confirmation bias],oneselectively gathers,orgivesundueweightto,evidence thatsupports one'sposition whileneglecting togather,ordiscounting,evidence thatwouldtellagainst it.“

Nickerson,R.S.(1998).Confirmationbias:Aubiquitousphenomenoninmanyguises.ReviewofGeneralPsychology,2(2),175.

EEtthhiiccaalliissssuueessffoorrffoorreennssiiccppssyycchhoollooggiissttss•Forensicpractitioners…

• striveforaccuracy,,fairness,andindependence.

• strivetotreatallparticipantsandweighalldata,opinions,andrivalhypothesesimpartially.

• strivetobeunbiasedandimpartial

SGFP, Guidelines 1.01and 1.02(APA, American Psychologist, January, 2013

Ethicalissues

"Whenpsychiatristsfunctionasexpertsinthelegalprocess,theyshouldadheretotheprincipleofhonestyandshouldstriveforobjectivity.Althoughtheymayberetainedbyonepartytoacivilorcriminalmatter,psychiatristsshouldadheretotheseprincipleswhenconductingevaluations,applyingclinicaldatatolegalcriteria,andexpressingopinions."

Ethical Guidelines forthePractice ofForensic Psychiatry, 2005, www.aapl.org. Section IV

EExxaammpplleessooffrreeaall--wwoorrllddffoorreennssiiccBBiiaass

ØFingerprintanalysisØDNAadmixtureØHairAnalysis

ExamplesofBiasFingerprintanalysis

Whenintentionallyprovidedwithmisleadingcontextual information(e.g.,confession),fingerprintanalystschangedtheirpriorclassification80%ofthetime.

◦ Itiel E.Dror,DavidCharlton,&AilsaE.Péron.(2006).Contextualinformationrendersexpertsvulnerabletomakingerroneousidentifications.ForensicScienceInternational,156(1),74–78.

ExamplesofBiasDNAAdmixture

17NorthAmericanexpertDNAexaminerswereaskedfortheirinterpretationofdatafromanadjudicatedcriminalcase,theyproducedinconsistent interpretations.Themajorityof'contextfree'expertsdisagreedwiththelaboratory'spre-trialconclusions,suggestingtheextraneouscontextofthecriminalcasemayhaveinfluencedtheinterpretationoftheDNAevidence

◦ SubjectivityandbiasinforensicDNAmixtureinterpretation.IEDror,GHampikian - Science&Justice,2011

ExamplesofBiasForensicAnthropologyInassessmentofsex

31% oftheparticipants inthecontrolgroupconcludedthattheskeleton remainsweremale.

72% concludedthattheremainsweremaleinthegroupthatreceivedcontextualinformationthattheremainsweremale.

0%oftheparticipantsconcludedthattheremainsweremale intheparticipantgroupwherethecontextwasthattheremainswereofafemale.

ExamplesofBiasHairAnalysis

Of28examinerswiththeFBILaboratory’smicroscopichaircomparisonunit,26overstatedforensicmatchesinwaysthatfavoredprosecutors inmorethan95percentofthe268trials

JURISPRUDENCE THE LAW, LAWYERS, AND THE COURT.

APRIL 22 2015 5:09 PM

Pseudoscience in the Witness BoxThe FBI faked an entire field of forensic science.By Dahlia Lithwick

The Washington Post reported that flawed forensic hair matches might have led to possibly hundreds ofwrongful convictions for rape, murder, and other violent crimes.

Photo by Victorburnside/Thinkstock

For more stories like this, like Slate on Facebook and follow us on Twitter.

The Washington Post published a story so horrifying this weekend that it wouldstop your breath: “The Justice Department and FBI have formally acknowledged thatnearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all

THE FBI FAKED AN ENTIRE FIELD OF FORENSIC SCIENCE

Biasispervasive.Itaffectsevery,andall,aspectsoflife

DAYTODAY LIFE

HistoryWarfarePolitics

CLINICALMEDICINE

FORENSIC PRACTICE

Ex-British PM Tony Blair apologizes for Iraq War 'mistakes'. UUSSAATTOODDAAYY1100..2255..22001155

• "Icansay thatIapologize forthe factthattheintelligence wereceived waswrong because, eventhough he(-------) had used chemical weapons extensively againsthis own people, againstothers,theprogram intheform thatwethought itwasdid notexist inthewaythatwethought.”

HistoryWarfarePoliticsCLINICALME

FORENSICPRACTICE

DICINEHowdifficultitistoacknowledgeerror!

Illustrationofconfirmationandblindspotbiases.

Normal pre-flight Flak pattern

• Statistical analysis reveals certainflak distributionpatterns(asabove)• Military analysts ask forextraarmoradded towhereholes clustered

• Wald challenges this andrecommends “considerjusttheopposite”– do notaddadditional protection tothose areas

Ifyou dothen itwillincrease your bomber loss ratio,notreduceit.

HistoryWarfare

BBeeccaauussee

• You arelimiting your observations tothose planes thatsurvived thebombing raidand returnedhome

• Those thatplunged and werelostdid not return, because theywerehitin thenon-pattern areas.They weredestroyed

• Therefore, reinforce those areasnot shown inthereturning planes.

HistoryWarfare

SurvivorshipBias

• Focusingonavailable evidence only• Ignoringmissing evidence• Notseeking contraryevidence

• Thosewhosurvived(returnedfromaraid).• Thosewhoprayedandsurvivedashipwreck• Eventhoughwepredictrisk,wedonotknowoutcomesafterrelease

DAYTO DAYLIFE

History

History is so indifferently rich that a case for almost any conclusion from it can be made by a selection of instances.Durant and Durant, The Lessons of History, 1968

The human understanding when it has once adopted an opinion (either as being the received opinion or as being agreeable to itself) draws all things else to support and agree with it.Francis Bacon c.1620

Cognitiveerrorshavebeenwithusforalongtime.HistoryPolitics

In the last analysis, we see only what we are ready to see, what we have been taught to see. We eliminate and ignore everything that is not part of our prejudice.Charcot. C. 1870s

SomeoftheCommonlyEncounteredBiasesinClinicalArena

• ConfirmationBias

• Retrospective(hindsight)Bias

• AllegianceBias

• IgnoringPrevalenceBias(Base-ratebias)

IIlllluussttrraattiivveeccaasseess

Multiple biases

• 32yr F.Ltarm tingling,M.S. incousin,domesticstress,MRIsmallparietal T2hyperdense spot.IVPrednisolone.Better.

• 6wks later,tinglyhands,facial numbness,thickspeech.MRInochange.Dx:worseningMS.

• Air ambulancetotertiary center.

AAnnootthheerriilllluussttrraattiivveeCCaasseeRReettrroossppeeccttiivvee BBiiaass

Setting50yrM.newonsetlowbackpainafewmonths.Active,healthy,overachiever.Normal exam.“ChoosingWisely” physiciandidnotimage.SymptomaticRx.Come forf/up.7days

Outcome

Potential Trial

Felt sowell, chosetowait. …………..

3wks lateracuteurinaryretention,catheterin,gaittroubleMultiple spinemets.

Defence:Followed guideline, Pt.failed tokeepf/up.No“redflags.”Whereispersonalresponsibility?Decision made under availableconditions.Outcome:yetunknown.

Guidelines acknowledgeexceptions,Physicianmusthaveimagedatfirstcontactinthecontextofa“nevercomplains” patient.Retrospectiveopinion,biasprone.Outcome:known

AllegiancebiasAreForensicExpertsBiasedbytheSideThatRetainedThem?

(Murrie, Boccaccini, Guarnera, &Rufino, 2013)

AAlllleeggiiaanncceebbiiaass•Subjectsbelieved theywerehiredbyeither

• defenseor• prosecution.

Examinedtworiskassessments: PCL-R and STATIC-99R

•Clearpatternofratingsemergedconsistentwithretainingside(uptod=.84)insomecases.

(Murrie, Boccaccini, Guarnera, &Rufino, 2013)

Confidenceandaccuracy•Confidence andaccuracyinassessments ofshort-termriskspresented byforensicpsychiatricpatients (Desmarais, Nicholls, Read, &Brink, 2010)

•Clinical vs.statistical prediction: “Adjusting”estimates ofriskontheSTATIC-99basedonexternal factorstypicallymakes predictionLESSaccurate.• (DeClue, 2013); (Hanson, Helmus, &Harris, 2015)

•Confidence notclosely associated withaccuracyincardiacdiagnostic evaluation(Cavalcanti &Sibbald,2014).

•But,see(Douglas&Ogloff,2003)

Wearefarbetteratrecognizingbiasinothers,ratherthanourselves

Neal, T.M.S.,&Brodsky, S.(n.d.).Forensic psychologists’ perceptions ofbias and potential correction strategies inforensicmentalhealth evaluations. Psychology, Public Policy, and Law.

TThhee““BBlliinnddSSppoott””BBiiaass

Neal,T.M.S.,&Brodsky,S.(n.d.).Forensicpsychologists’perceptionsofbiasandpotentialcorrectionstrategiesinforensicmentalhealthevaluations.Psychology,PublicPolicy,andLaw.

ü Expertsweresurveyed,andfound:

ü Mostrecognizedbiasinotherexperts,butbelievedtheyandtheircolleagueswereunbiased.

v Expertsusedmanyapproachestoavoidbias,butmosthadbeenshowntobeineffective,including:

v Introspectionofpersonalbiases

v Reflection oncase

v Loyaltytodoinga“goodjob”

Base-ratebias

Empirical evidence thatdisease prevalencemayaffecttheperformanceofdiagnostic tests withanimplicit threshold:across-sectional study.

Willis, B.H.BMJOpen, 2(1), (2012).

Walters, Kroner, DeMatteo, &Locklair, 2014

Whatnext?

• Isbiasinevitable and ubiquitous?• Isitanormal operating characteristic of ourbrains?

• Isitevenpossibletoescapebias?

• We cancertainly aspire• ForAwareness thatbias exists in usand others, and• ToMinimize bias

WhatisdebiasingDoesitwork

Will itworkinallbias inducingsituationsWhatistheexperience sofarinclinicalcare

Will itworkinforensicarena

DDeebbiiaassiinngg eevviiddeennccee rreevviieewwExperiment orReview Method Findings Suggestions orConclusions

Lichtenfiled2009.Narrative,experientialessay.Emphasis:conf.biasanddebiasingmethods.

Extensive cataloguingof cognitivebiases Paucityof research andmixedevidenceoneff icacyof debiasing.

Graber2012Narrativereviewof141articlesoninterventionstoreducediagnosticerrors Lack real-worldsituations. Afewhelpfulwithtrainees.Fieldisimmature.

Sherbino2014(n-191)Emergencymedrotationstudentsusingcomputerbasedcases.Aim:reducediagnosticerrors.

Educationalinterventions (cognitiveforcingstrategies) toreduceerrorsfailed.

Stiegler2014Reviewofdecisionmakingmodelsandnonrational(notlogicalorpurelystatistically derived)factorsinanesthesiology.Strategiestorecognizeandrecoverfromerrors.

Cognitiveshortcuts,preferencesandemotionsinf luencerealworlddecisions.

Routinestrategiestomodifynonrationaldecisionfactorsdonothaveunequivocalsupport.

Normal2014(n=214)Tworesidentcohorts:onespeedyothercareful,reflectiveEachcomparedfordiagnosticaccuracy.

Noincreaseinaccuracy fortheref lectivecohort.

Blumenthal-Barby2015Revieworiginal,peerreviewedempiricalstudiesoncognitivebiasesandheuristicsinmedicaldecisionmaking(1980and2013,n-213)

Studiesonbiasandheuristicsof medicaldecisionmakingshouldfocusmoreonactualratherthanhypotheticalsituations. Patientsarestudiedmoreoftenthanmedicalpersonnel.Terminology usedincognitivebiasstudiesarenotstandardized.

Balogh2015.NAPBook.Reviewofevidenceaboutdiagnosticerrors,andrecommendationsforimprovement.

Mostpersonsexperience1diagnosticerror,andsomesufferseriousconsequences.Failedheuristicsandcognitivebiasesmayleadtoerrors.Effectsof debiasing,whiledebatable,maybehelpfulincertainsituationsasemergencyrooms.

Moreresearchinneededinthisarea.

Monteiro201547medicalresidents.Reflectandrevisediagnosesfor16cases.

Unstructuredref lectionofcaseshadasmallbenef it. Diagnosticperformanceismodulatedbyexperienceandknowledge

Smith,Black2015.Debiasingworkshopfor19familymedicineresidents.Objective:toincreaseresidentsawarenessoftheircognitivebiasleadingtomisdiagnosis.

Inspiteof positivetrends,statisticallysignif icantimprovementsdidnotoccur.

Educating traineephysiciansinrisksof biasandmisdiagnosismaynotbeeffective.

DDeebbiiaassiinnggEEvviiddeenncceeRReevviieeww

WWhhaattddooeessiitttteelllluuss??

Therearemorestudies incognitivepsychology field, and farfewer inclinicalmedicine

Clinical &cognitive psychology fields need tocollaborate

Field needs tomature.Need real-life, not hypotheticalsituations

Some positive trends.Checklist inICUand surgeryCurriculum addition among trainees

Negativeresults also evident

Terminology notstandardized

GGeenneerraallDDeebbiiaassiinngg aaddvviissee..

Typeofcognitiveerror

Debiasing,afterawarenessandacceptanceofneedforchange,

Confirmationbias Activelyseekdisconfirmingdata; isthereevidencecontradictingyourhypothesis?Considertheoppositeofyour diagnosis orsummation;disengage, when possible, from dual roleof treatingand being anexpertwitness

Hindsight/Retrospectivebias

Whilerenderinganopinion,considerwhatyourdecision(diagnosis)wouldhavebeenifyouwereblindtotheultimateoutcome

Baseratebias Beforeselectingcasespecificdiagnosisorconclusions,findouthowcommonthatdiagnosisis (baserates)amongthepossibilitiesinthatpopulation

Modified from Satya-Murti S, Lockhart J. Recognizing and reducing cognitive bias in clinical and forensic neurology. Neurol Clin Pract. 2015;5(5):389–396.

De-biasingstrategiesOr,AnounceofBiaspreventionisworthapoundofDe-Biasing“Cure.”◦Biaspreventionshouldbeprimarygoal,withde-biasingtechniquesbeingsecondary.

◦Usede-biasingtechniques whenprevention isunavailable.

Biaspreventionstrategy:“Linearsequentialunmasking”

LinearSequentialUnmasking(LSU)ApproachforMinimizingCognitiveBiasinForensicDecisionMaking(Dror etal.,2015)

Level1:TraceEvidence

Level2:Referencematerials

Level3:Caseinformation

Level4:"Baserate"expectations

Level5:Organizationalandculturalfactors

BiasPreventionstrategiesControlpotentiallybiasinginformationfromthebeginningReviewmostvaliddatafirst,beforepotentiallybiased(prematureclosure,dxmomentum)•Avoid“diagnosticmomentum”byformingatentativeopinionBEFOREreadingotherexperts’conclusions• Thatway,youcanpinpointtheevidencethatchangedyouropinion

Sensitizeyourselftopotentialbiases(e.g.,Jenkins&Youngstrom,2016)

GeneralDebiasing advise.Typeofcognitiveerror

Debiasing,afterawarenessandacceptanceofneedforchange,

Confirmationbias Activelyseekdisconfirmingdata;isthereevidencecontradictingyourhypothesis?Considertheoppositeofyourdiagnosisorsummation;disengage,whenpossible,fromdualroleoftreatingandbeinganexpertwitness

Baseratebias Beforeselectingcasespecificdiagnosisorconclusions,findouthowcommonthatdiagnosisis(baserates)amongthepossibilitiesinthatpopulation

Hindsight/Retrospectivebias

Whilerenderinganopinion,considerwhatyourdecisions(diagnosis)wouldhavebeenifyouwereblindtotheultimateoutcome

Modified from Satya-Murti S, Lockhart J. Recognizing and reducing cognitive bias in clinical and forensic neurology. Neurol Clin Pract. 2015;5(5):389–396.

De-BiasingTechniques•Requireseffortfulprocessing(notmerely“reflection”).E.g.,activelyseekingoutevidencethatisinconsistentwithyourconclusion

•Usechecklists/objectivemeasureswhereavailable(e.g.,MacCAT-CA,ECSTinCSTevals).Witt(2010),MACSTChecklist,Skeem (1989)paperallincludedinmaterials

•Consultwithacolleaguewhohasminimal info,andwhoisnot afraidtodisagreewithyou.

RReeccooggnniittiioonnooffBBiiaassmmaakkeessyyoouuaabbeetttteerreexxppeerrttWWiittnneessss::JJooeellDDvvoosskkiinnvIfyouonlyconsideronesideofthequestion,youarelesslikelytobeseenasacrediblewitness.

v“Thereisnosuchthingas‘THETRUTH.’“(onlyobjectivefactsandyouropinionabout them

vYouareevidence(likeamaggot,butnicer)

v“Showyourwork” letthereaderseetheevidencefor(andagainst)yourconclusion

vContinuallyaskyourself,“HowdoIknowthat?”

vRemember,“IT’sNOTABOUTYOU!”

Thankyouforlistening toour

SatySatya-Murti&JosephLockhart

unbiasedpresentation

LINKS TO ADDITIONAL RESOURCES

These are links to Dropbox files of articles for checklists to use in forensic report-writing:

1. Checklist for CST reports from MAhttps://www.dropbox.com/s/rz1jhc6trqd1mns/DubeCRreportwritingguidelinesrevised2002.pdf?dl=02. Skeem - Logic and relaibility of CST Evaluations (valuable tool for examining your reasoning)https://www.dropbox.com/s/cvovm4ihndudlbo/Skeem%20et%20al.%20-%201998%20-%20Logic%20and%20reliability%20of%20evaluations%20of%20competence%20copy.pdf?dl=03. Witt - General checklist for Forensic evaluationshttps://www.dropbox.com/s/rgfdaeqcj2vlyav/Witt%2C%20Phil%20-%20Forensic%20Psychology%20Report%20Checklist.pdf%20copy.pdf?dl=04. Grisso-Typical errors in forensic evaluationshttps://www.dropbox.com/s/jmcikysz4kb23vf/Grisso%25202010-2%5B1%5D.pdf?dl=0

Contact: Jerry Lockhart (josephjlockhart@gmail.com)