Reasonable Doubt Uncertainty in Education Science and Law
-
Upload
diego-alonso-collantes -
Category
Documents
-
view
213 -
download
0
description
Transcript of Reasonable Doubt Uncertainty in Education Science and Law
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 1 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
UniversityPressScholarshipOnlineBritishAcademyScholarshipOnline
Evidence,InferenceandEnquiryPhilipDawid,WilliamTwining,andMimiVasilaki
Printpublicationdate:2011PrintISBN-13:9780197264843PublishedtoBritishAcademyScholarshipOnline:January2013DOI:10.5871/bacad/9780197264843.001.0001
ReasonableDoubt:UncertaintyinEducation,ScienceandLawTONYGARDNER-MEDWIN
DOI:10.5871/bacad/9780197264843.003.0017
AbstractandKeywords
Thischapterchampionstheelicitationanduseofprobabilisticmeasuresofuncertainty.Itcomparestwodifferentviews(roughlycorresponding,respectively,totheBayesianandfrequentistapproachestostatisticalinference)astohowprobabilitiesshouldbeusedasevidence.Itarguesthat,whiletheformermaybealogicalideal,thelattermaybemoreappropriateinthespecialcircumstancesofthecourtroom.
Keywords:uncertainty,education,probabilities,evidence,statisticalinference
AbstractTheuseofevidencetoresolveuncertaintiesiskeytomanyendeavours,mostconspicuouslyscienceandlaw.Despitethis,thelogicofuncertaintyisseldomtaughtexplicitly,andoftenseemsmisunderstood.Traditionaleducationalpracticeevenfailstoencouragestudentstoidentifyuncertaintywhentheyexpress
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 2 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
knowledge,thoughmarkschemesthatrewardtheidentificationofreliableanduncertainresponseshavelongbeenshowntoencouragemoreinsightfulunderstanding.Inourinformation-richsocietytheabilitytoidentifyuncertaintyisoftenmoreimportantthanthepossessionofknowledgeitself.
Inbothscienceandlawtherearefundamentallydifferentkindsofuncertaintyatissue.Thereisuncertaintywhetheraparticularhypothesisiscorrect,andthereisuncertaintyaboutobservabledatathatmaybegeneratedifaparticularhypothesisiscorrect.Bothareexpressedintermsofprobabilities.Eachhasitsowndomainofapplicationanditsownlogic,buttheinterrelationshipiscomplexandsometimesmisunderstood.Hypothesisprobabilitiesarealwaysopentoerrorthroughpossiblefailuretotakeaccountofrealisticalternatives,whiletheproperinferencesthatcanbedrawnfromdataprobabilities(ofteninthecontextofsignificancetesting)arequitelimitedandeasilyover-interpreted.
Whenconsideringthesetwokindsofprobabilityinacourtoflawitispossibletointerpretthephrasereasonabledoubtindifferentways.Itcanbeseenasaddressingdatauncertainty:whethersuchincriminatingevidencemightwithreasonableprobabilityarisetoconfrontaninnocentperson.Or(themoreconventionalview)itcanbeseenassomesortofthresholdlevelontheprobabilitythatthedefendantisguilty(ahypothesisprobability).Eachtypicallyinvolveselementsofsubjectivejudgement,butfewerissuesanduncertaintiesarisewhenconsideringthedataprobabilityanditisarguedthatthisisoftenthemorecriticalandproperissueforajurytoaddress.ThishasparticularrepercussionsforcasesinvolvingidentificationofasuspectthroughtrawlofaDNAorotherdatabase.
(p.466) UncertaintyandmisconceptionineducationalassessmentASAUNIVERSITYTEACHER(inmedicalscience)Ihavetriedtoraisestudentsawarenessofuncertaintiesintheirownknowledge.Inconventionaleducationalassessmentstudentsareoftenmotivatedtohideuncertainties.Thisisperverse,becauseitisobviousthatitisagoodthingtobeabletodistinguishbetweenreliableandunreliableaspectsofonesknowledge.Studentsshouldinfactberewardedforacknowledginguncertaintiesthattheyhave.Byrewardingthem,forexamplewithsimplebutcarefullydesignedmarkschemes,theycanbeencouragedtoreflectonthenatureofanydoubts,andontheevidencethattheymaybeabletobringtobeartoresolvethesedoubts.Insteadofdoingthis,conventionalmarkingusuallyencouragesstudentstoblufftheirwaythrough:ittreatsluckyguessesinthesamewayaswelljustifiedknowledge,andfirmmisconceptionsasnoworsethanacknowledgedignorance.Theresultisago-for-itcultureinwhichdecisionsmaybetakeninthelightofamarginalorsuperficialpreferenceforsomeoption,withlittlefurtherthought.
Thedangersofsuchbehaviourareveryobviousinafieldsuchasmedicine,wherelivescanbelostthroughreluctancetoacknowledgeuncertainty.Eveninordinarydiscoursehowever,itisasignofweaknesstoshyfromproperawarenessandacknowledgementofonesuncertainties.Itisdisturbing,whentalkingtostudentsfamiliarwithessaywritingwhereyoumightthinkthediscussionofuncertaintieswouldbeparamountthatthey
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 3 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
oftensaytheywouldnotnormallymentiondoubtsaboutafactorargumenttheyaretemptedtoinclude.Anyonewhohasmarkedexamessaysknowsthatinpracticeaqualification,perhapsaquestionmarkinthemargin,willmakelittledifferencetoonesjudgementifthepointmadeiscorrect,whileitmakesabigdifferenceifuncertaintyisacknowledgedaboutsomethingincorrect.
Itmayatfirstseemparadoxicaltorewardsomeoneforacknowledginguncertainty.PerhapsitrunscountertosocietysmodernguidelinesforHowtogetoninpolitics,business,andmaybesometimesevenscienceorlaw.Buttherewardsforcautioninthefaceofuncertaintyarefundamentaltobiologicalsurvival.Childrenandyounganimalslearntheselessonsoftenthroughgames,anditisnocoincidencethattherelevantmathematicscomesundertheheadingGameTheory(vonNeumannandMorgenstern1944).Of(p.467) coursecertaintyalsobringsitsownrewardsjustaslongasitiscertaintyaboutsomethingthatiscorrect.Topursueacricketinganalogy,thedecisiontotrytohitaballforsixisfine,solongasoneisprettysureonecansucceed:thepriceoffailuretendstobehigh.Ifoneisuncertainofsuccess,thenamoremodeststrokewillonaveragebebetterrewarded.Exactlythesameprincipleisemployedinthecertainty-basedmarkschemeweemployforself-testsatUCLandImperialCollegeinLondon(Gardner-Medwin1995,2006).Acknowledginguncertainty(67%estimatedprobabilityofbeingcorrect)gives1markforacorrectanswerand0ifincorrect.Claimingahighlevelofcertainty(80%)gives3marksor6,whileanintermediatelevelgives2marksor2.Thisisaproperormotivatingmarkscheme,inthesensethatastudentalwaysexpectstogainbygivinganaccurateindicationofhowreliablehe/shethinksthechosenansweris(Dawid1986).Withthisschemethereisnobenefittobegainedbythestudenttryingtoplaythesystem:pretendingtobeeithermoreorlessconfidentthanhe/shereallyis.Itisremarkablehowreadilystudentstaketothisscheme,appreciatingthegoodsenseandutilitarianvalueofsettlingformodestrewardsandpenaltieswhenunsure.
Uncertaintyabouttheexpressionofknowledgeorunderstandingisanexampleofuncertaintyaboutideas.Muchofeducationaltestingrevolvesaroundideasthataregenerallyaccepted(atleastwithinanagreedframework)asdefinitelyeithertrueorfalse,thoughstudentswithpartialknowledgemayassignprobabilitieslessthan1andthosewithseriousmisconceptionsmayassignverylowprobabilitiestowhatiscorrect.Thereareofcoursemanyelementstoeducationalexperiencewherethereisnocertaintyandthetaskistogenerateordiscussideasthatareneitherknownnoragreedtobeobjectivelycorrectorincorrect.Herewecomeclosertotheissuesthatariseinscientificresearchandincourtsoflaw.
Uncertaintyaboutideas:thedrivingforceinscienceScientistsworkwithtwofundamentallydifferentkindsofuncertainty,thoughtheydontalwaysdistinguishthemveryclearly.Thefirstisanintrinsicallysubjectiveprobability:Howcertainisitthataparticularideaorhypothesisiscorrect?Thisisobviouslyfundamentalwithinscience,buttheanswerstosuchquestionsarealwayssubjectiveinthesensethattheymaydifferevenbetweenwellinformedandintelligentindividuals.Bygainingadditionalevidenceonemayshiftsuchsubjectiveprobabilitiesinwaysthatall
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 4 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
agreearerationalandoccasionallyevenquantifiable.Indeed,asPopper(1959)pointedout,onecansometimesdemonstratethatanideaiscertainlywrong,thoughneverthatitis(p.468) certainlyrightasageneraltruth.Differentlevelsofscepticismanduncertaintyamongstindividuals,aboutwhatiscorrect,mayneverbeeliminated.Nevertheless,thereareofcoursegreatswathesofscientificconclusionsthatareregarded,bymoreorlessallwhohavestudiedthem,asessentiallycertain.Itisthepowerofsciencethattheaccumulationofevidence,givenenoughtimeandeffort,tendstobecomeoverwhelmingononesideortheotherofanyargument.However,evidencedoesnotalwaysdiminishuncertainty,andevenfirmlyembeddedideascanoccasionallybechallengedoroverturned,evenbysimplepiecesofevidence.Uncertaintyaboutideasisthemotivationforprogressinscience,despiteitsnebulousandlargelyunquantifiablenature.Itdrivesthetestingofpredictionsandthedevisingofalternativetheoriesandexperimentalchallenges.
Darwinstheoryofnaturalselectionisaprimeexampleofanideathatwasconceivedwithahugeelementofuncertainty,notleastinDarwinsmind.Darwinsgreatsuccesswasthatheamassedavastamountofevidenceinitsfavourbeforegoingpublic:enoughevidencetodiminishhisownuncertaintyandtoshaketheconvictionofmanysceptics.Thereneverwas(andneverwillbe)atimewhenDarwinstheoryistotallycertain,beyondanymeaningfulthreshold.Ofcourse,nearlyeveryscientistbelieves,asIcertainlydo,inthistheory.Darwinsideaisasfirmlyembeddedasanyinscience,thankstotheprogressiveelucidationofbiologicalmechanismsandtheaccumulationofdatathatareall(sofarasIknow)consistentwiththecoreelementsandpredictionsofthistheory.Buttherecanbenocertaintythatnewideasandobservationscouldnotchangeit.AscientistsnaturalresponsetothequestionIsDarwinstheorybeyondreasonabledoubt?istosayWhatdoubt?Letsseeifwecandeviseatestforanyideasabouthowitmaybewrong.Scientistslovecoherentscepticism,justsolongastheideasaretestable.Thehistoryofscienceteachesthemtoregarddogmaticbeliefsaspointlessandarrogant.Thestrengthofthechieftenetsofscienceisnotthattheyarecertain,butthattooverturnthemwouldrequirenotonlyobservationsinconsistentwiththem,butalsoanewexplanationforalltheevidencethathashithertoappearedtosupportthem.Darwinachievedthisinrelationtothenotionofdivinecreation,asdidEinsteinwithclassicalmechanics.Scientistsseldomofcourseachievesuchdramaticrevolutionaryimpact;butevenwhenthepursuitofuncertaintydoesntoverturnorstrengthenexistingideas,itoftenthrowslightonfacetsthatwerepreviouslyunclear.
(p.469) UncertaintyaboutexpecteddataThesecondkindofuncertaintyisconceptuallymorestraightforwardandquantitative,thoughoftenunderpinnedbycomplexmathematics.Thisisuncertaintyaboutthedatathatwillariseinanexperiment,giventhattherelevantmechanismsandprinciplesareeitherfullyunderstoodorfullydefinedbyahypotheticalmodel.Statisticalanalysisofsuchuncertaintyispartofanyscientifictraining(andthebaneofmanystudents!).Itgivesaframeworkforevaluationofnewresults,sinceanovelclaimisoflittlevalueifthenoveltyorinterestoftheevidencecouldquitelikelyhavearisenwithconventionaloruninterestingassumptions.Thecommonestformatisthatofsignificancetesting,initiated
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 5 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
byFisher(1925).TheassumptionsonwhichasignificancetestisbasedaredescribedasanullhypothesisH0.Apointofpotentialinterestisencapsulatedinastatisticfromtheobservations,quantifyingadifferencefromthemeanormedianvalueexpectedonH0.Theoutcomeofthetestistheprobability(P-value)basedonH0that,withtheproceduresadoptedintheresearch,atleastaslargeavaluewouldbeobserved.
SincetheuseandmisuseofsignificancetestsandP-valueshasahugeandcontroversialliterature(seeforexampleveryapproachablereviewsbyRoyall(1997)andSenn(2003)),Ishallpresentherejustapersonalperspectiveonsomeoftheseissues.Afundamentalpoint,clearIhopeinthedescriptionabove,isthataP-valueisaprobabilityofobservingdata,conditionalonaspecifichypothesis.Itdoesnotexpressuncertaintyabouthypotheses.P-valuesarehugelyvaluableinsciencebecausetheyquantifythedegreetowhichdataareconsistentwithsomeconventionalorpostulatedidea.PerhapsthemostusefulroletheyperformiswhentheP-valueislargeandtheresultisnot-significant:thismeansthatthepointofinterestinthedatawouldbequitelikelytoariseevenifH0istrue,sothereislittlereasontopayattentiontothisaspectofthedataasanykindofchallengetoH0.Onecanconcludethisquitestraightforwardly,withoutanyconsiderationofalternativehypothesesortherathernebulousbusiness,discussedinthelastsection,ofaddressingtheprobabilitiesthathypothesesmaybecorrect.
AhazardintheinterpretationofP-valuesarisesfromterminologyoftenusedtoexpressconclusionsbasedonthem.LowP-valuesareoftensaidtojustifyrejectionofH0ataparticularsignificancelevel.Thisisonlytrueinaveryrestrictedformalsense,whenaP-valueisusedasadecisioncriterionbetweentwoactionsthatwouldbeappropriateifthereweresolidreasonstoacceptorrejectH0.ForexampleonemightdecidetoignorethedatabecauseitisconsistentwithH0,orstartresearchingalternativehypothesesifthedatawouldbesurprisingonthebasisofH0.Iftherearealternativehypotheses(p.470) alreadyformulated,withsomebasisforassigningthemrelativeprobabilities,thenaresultwithalowP-valuedoesntnecessarilyevenargueagainstH0,letalonejustifyrejectingit.AninterestingresultmaybesoextremethatitisveryunlikelyonH0,butitmaybeevenlesslikelyonwhateveralternativehypothesesareconsideredplausible.Asanexample,supposethattentossesofacoinyieldnineheadsandonetail.Doesthisresult,surprisinglyfarfrom50:50(P-value=2percent)argueforrejectionofahypothesis(H0)thatthisisafaircoin?Iftheonlyalternativesseemtobethatacoinmighthavetwoheadsortwotails,thenthedataclearlyshowsbothofthesefalse,supportingH0.However,ifoneenvisagesthepossibilitythatacoinmightbebiassedsoastobemorelikelytolandoneway,andconsidersthisplausible,thenthedatalendmoresupporttothisideathantoH0.WhetheroneendsupconsideringH0moreorlesslikelythanattheoutsetdependsontheinitialprobabilitiesoneassignstoanyofthese(andpossiblyother)ideasonespriorprobabilities.Inscience,especiallyatthefrontiers,thealternativestoawell-formulatedhypothesisareoftensimplyamatterofspeculation.InthatsituationallonecansensiblysayaboutdatawithalowP-valueisthattheparameterofinterestissurprisinglyfarfromexpectationonthebasisofH0(orsimilarhypotheses)andthatotherhypotheses,notnecessarilyveryplausibleoreventhoughtof,couldhaverendereditlesssurprising.
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 6 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
TheborderlinebetweenuncertaintyabouthypothesesanddataAusefulextensionofsignificancetestsisthecalculationofso-calledconfidencelimits:arangeofnullhypothesesthatwouldyieldP-valuesaboveaspecifiedlevelforaresultofinterestinthedatainotherwords,theywouldmaketheresultunsurprising.Thispracticehelpstoclarifyhowusefultheobservationsare:howwelltheyservetodiscriminatebetweendifferentvaluesofaparameterwithinahypotheticalmodel;butitstillsaysnothingabouttheprobabilitythatoneshouldassigntothecorrectnessofanyofthesemodels.Thecommonlyusedterminologyisagain,inmyview,misleadingbecausetheterm95percentconfidencelimitsforaparameterinamodelcaneasilybeunderstoodtomeanthatthereisa95percentprobabilitythatthetrueparameterlieswithintheselimits.Thisissimplynotso,asisevidentifyouthinkaboutanexperimentthattriestomeasureaneffectthatyouconsidertobealmostcertainlynon-existent(perhapsextrasensoryperceptionorhomeopathy).Whatevertheprotocolfortheexperiment,youshouldexpecton5percentofoccasionsthataresultwillbeobtainedforwhichanileffectisoutsidethecalculatedconfidencelimits.Ifyouhappentoexperienceoneof(p.471) theseresultsitwouldobviouslybeinappropriatetoadoptabeliefthatthereisa95percentprobabilitythatthepostulatedeffectisreal.Perhapsconsonancelimits(KempthorneandFolks1971)wouldbetterconveythetruemeaning,whichisarangeofmodelsforwhichthedatawouldberelativelycommonplacefallingwithinthe95percentmostlikelyoutcomes.
Ingeneralitisimportanttomakeacategorydistinctionbetweentheprobabilitythatdatawillariseandtheprobabilitythatahypothesisaboutthegenerationofdataiscorrect.Sometimes,however,thereisasimplerelationshipbetweenthetwo.Forexample,anexperimentmayestimate(bysamplemeasurements)themeanvalueofaparameterthatisconsideredattheoutsetequallylikelytotakeanyofawiderangeofvalues:perhapssomecharacteristicofanewlyencounteredsubstance,objectorspecies.Then(withafew,oftenjustifiable,assumptions)1theposteriorprobabilitiesofdifferenthypothesesaboutthetruemeanarethesameasthedataprobabilitiesthatwouldbecalculatedforsamplemeans,onthehypothesisthatthetruemeaniswhatwasactuallymeasured.Returningtoterminology,thismeansthatwhennohypothesisinitiallyhaspreferredstatus,thelimitscautiouslydescribedaboveas95%consonancelimitsaretruly95percentconfidencelimitstheyidentifyarangethatonecanbefairlyconfidentincludesthetruemean(with95percentprobabilitybasedononlythesedata).Exampleslikethiswhereonecanconvertdirectlyfromadataprobabilitytoahypothesisprobabilityarenotrare,buttheyleadtoconfusioniftheirlimitationsarenotproperlytakenintoaccount.
Whenalternativehypothesesarewelldefinedanddohavespecialstatusinanexperiment,forexamplerivalcausalexplanationsofthedata,thenupdatingofprobabilitiesforthealternativesrequiresBayesiananalysis(Lindley1972).Aswesawabove(inthecoinexample)attemptstomakesuchinferencesusingP-valuescanbequitemisleadingsinceanunlikelyresultonahypothesisH0mayactuallybeevidenceinfavourofH0.Bayesiananalysisinvolvesidentifyingpriorprobabilitiesforthevarioushypothesesandupdatingtheseinthelightofthedata,usinglikelihoodratios(therelativeprobabilitiesthatthedatawouldbeobservedonthedifferenthypotheses).Thetechnicalitiesneednot
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 7 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
concernushere,buttheprocedureisdemonstrablysuccessfulinsituationswherealternativehypotheseshaveclearpriorprobabilities.Evenwhenthisisnotthecase,andpriorprobabilitiesarelittlemorethanguesses,sufficientdatacansometimesleadtohighlyreliableinferences(seeforexample,(p.472) MacKay2003).Themainreservationisthatconclusionscanbewhollymisleadingifthecorrecthypothesishasbeenomittedfromconsiderationorassignedaninappropriateprobability.Ifthisisamajorrisk,asitcanofcoursebeatthefrontiersofresearch,thenthemorelimitedlogicbywhichsignificancetestschallengeindividualhypothesescanbeamorecomfortablebasisforscientificprogress.
AnimportantapplicationofBayesiananalysisisinmedicaldiagnosis.Hereonestartswithfairlyobjectiveprobabilitiesthatdifferenthypothesesmightbetrue,forexamplethatapersonwithapersistentcoughmayhaveeachofanumberofdifferentdiseases.Thentheseprobabilitiesareupdatedonthebasisofdatafromthepatientshistoryandtestscarriedout,usinginformationabouthowfrequentlythedatawouldariseinthepopulationforeachoftheconditions.Thisprocessmaysimplyoccurinthedoctorsmind,leadingeventuallyeithertoconfidenceaboutspecificdiagnosesortoresidualuncertainty.Alternatively,aBayesiancomputeralgorithmmayprocessthedatatogenerateexplicitposteriorprobabilities.Whichisbetterisdebatable,giventhepossibleunreliabilityofthedoctorscompilationofinferencessetagainsthisorherabilitytotakeintoaccountsubtleaspectsofthedataignoredbyacomputeralgorithm.Butthereisnodisputingtheappropriatenessofthisstrategytoresolvemedicaluncertaintiesandtohelpmakedecisions.
UncertaintyincriminaltrialsTheuncertaintiesinacriminaltrialhavesuperficialsimilaritiestotheproblemsofmedicaldiagnosis.Theendpointisadecisionthatturnsgradeduncertaintiesintospecificactions:whethertoconvictadefendant,orapplytreatmentforaspecificdiagnosis.Therelativeutilitiesofrightandwrongdecisionsmayinpartdeterminewhatlevelofdoubtisacceptableforafinaldecision.Inmedicinethisuncertaintyisnormallyexplicit,sincethepatientusuallymakesthefinaldecisionaboutwhethertoaccepttreatment,andmustthereforebeinformedofdoubtsandlikelyoutcomes.Inlaw,thecriterionforconviction(beyondreasonabledoubt)issubstantiallyopentoajurysinterpretation,withonlypartialknowledgeoftheutilitiesinvolvedsinceasentenceforconvictionmayonlybedeterminedatalaterstagebyajudge.Butthesearenotprofounddifferences.
Legalcasesoftenhavemanylayersofcomplexity,forexampleconcerningfacts,motives,identification,intentionandwitnesscredibility.Finaldecisionsmayhingeonacomplexsynthesisofmanytypesofdoubt(Cohen1977;Andersonetal.2005),anditisfarfromclearthatmathematicalapproaches(p.473) arehelpfulincomingtoappropriateconclusions.Alternativeconceptionsofprobability(forexampleCohensinductiveprobabilitybasedonqualitativeeliminativereasoning)maysometimesbemorerational(oratleastmoremanageableforajury)whendecidingwhetherfactsarebeyondreasonabledoubtinatrial.Suchissuesaremuchdebatedinthelegalliterature(seeforexampleTillersandGottfried2006)withcogentargumentsfrommanyperspectives.
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 8 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
However,thedebatethatfollowsnaturallyfromthediscussionhereonthenatureofuncertaintiesinscienceisnotaboutwhetherprobabilitiesshouldbequantifiedbutwhetherweshouldbedealingwithuncertaintiesabouthypothesesoraboutdata.Ideally,ofcourse,onewouldlikecourtdecisionstobebasedoncertaintyratherthanuncertainty,butthisisrarelythecase.Iftheevidencedoessupportcertainty,thenconcludingthatdefendantDdefinitelycommittedcrimeC(anassertionaboutahypothesisprobability)isequivalenttoconcludingthattheevidencecouldnothavearisenifDwereinnocent(anassertionaboutadataprobability):eachconclusionfollowsfromtheother.Butifuncertaintiesarepresent,thesebecomedifferentquestions.
Ihavearguedelsewhere(Gardner-Medwin2005)thatitcanberationalforajurytocometotheconclusionthatthehypothesisofguiltisveryprobablycorrect,whileacquittingonthegroundsthattheevidencecouldwithreasonablelikelihoodhavearisenforaninnocentperson.Itmayatfirstseemthattheseissuesareoppositesidesofthesamecoin,whichwouldmakemyassertioneitherparadoxicalormerelyamatterofsettingdifferentthresholdsforjudgingthetwoprobabilities.Iwillfirstsetoutacaricatureexample,basedoncasesthatfirstdrewmyinterest(asdiscussedbyDawid2002)toshowthatthisisnotthecase.
Supposemultipleinfantsinafamilyhavediedincircumstancesconsistentwitheithersuddeninfantdeathsyndrome(SIDS:araremedicalconditionthatleavesnospecificsignspostmortem)orinfanticidebythemother.Thismayreasonablyleadtosuspicionofcrime,andindeedcaseshavecometocourtwithlittlemoreinthewayofpertinentevidence.Itisobviouslyrelevanttoaskquestionsanalogoustothoseunderlyingtestsofsignificance:Howlikelyisitthatsuchevidencewouldariseinaninnocentfamilywithcomparablegenetic,medicalandsocio-economicbackground,andhowoftenwouldsuchcasesbeexpectedtoariseinaninnocentpopulation?Suchquestionscannotbeansweredwithprecision,butcompetentexpertsshouldbeabletogivereasonablerangesfortheanswers.UnfortunatelythesupposedlyexperttestimonyinrecentUKcaseswasnotcompetent,raisingseriousconcernsabouthowexperttestimonyshouldbevalidated(RoyalStatisticalSociety2002);butthisdoesnotdetractfromtherelevanceofproperanswers.Theconclusionmightbethattheriskofthedeathsthathavebroughtthismother(p.474) tocourtarisinginherfamilywithoutcrimewasatleastaprobabilityP,andthatsimilarmultipledeathsinfamilieswithnogreaterriskfactorsformultipleSIDSmightbeexpectedtoarisesomewhereintheUKonceeveryXmonths.Ajurymightreasonablydecidetoacquitsimplyonthebasisofsuchtestimony,justifyingtheirdecisionbysayingthatifjuriesconvictonsuchevidencetheymightberesponsibleforconvictinginnocentmothersattherateofoneeveryXmonthsintheUK,whichtheydeemunacceptable.Theacceptabilitycriterionisofcourseasubjectivematter:howmanyfalseconvictionsofthissortmightbeacceptableinagivenpopulationperyear,decadeorcenturyintheinterestsofjustice.Butitisnomoresubjectivethanwhatconstitutesreasonabledoubtonanyotherbasis.
Analternativeapproachistodecidesuchacasebasedontheprobabilitythatthehypothesisofguiltiscorrect,thatisthatthisdefendantisguiltywithinthemeaningofthe
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 9 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
law.Thisviewisoftenimplicitinthelegalliteratureandthemedia.ItwasclearlyexpressedintheletterfromtheRoyalStatisticalSociety(2002)totheLordChancellorintheUKconcerningthemultipleSIDScaseofRvs.SallyClark,andwasanalysedinrelationtothiscaseinmoredetailbyDawid(2002).Toquotetheletter:Twodeathsbymurdermaywellbeevenmoreunlikely[thantwodeathsbySIDS].Whatmattersistherelativelikelihoodofthedeathsundereachexplanation,notjusthowunlikelytheyareunderoneexplanation.Theimplication(madeexplicitbyDawid(2002)inSection2.3ofhispaper)isthatajuryshoulduseevidenceabouttheincidenceofthistypeofmurderinthepopulation,andconvictifthisissufficientlygreaterthanthatofSIDS,takingaccountofalltheknowncircumstances.Thisiscertainlyarationalwaytoinferprobabilityofguiltinlightoftheevidence,butithasuncomfortableconsequenceswhenusedasthebasisforconvictionoracquittal.Inparticular,itmeansthatadefendantwhowouldbeacquittedonthegroundsthattheevidenceisreasonablyconsistentwithinnocence(theargumentofthelastparagraph)mightfindherselfconvictedbecausesufficientlymanyotherpeoplehavecommittedthecrimeofwhichsheisaccused(Gardner-Medwin2005).Thisseemsethicallyimproperandprobablyunacceptableinlaw,becausethevoluntarycriminalactsandintentionsofpeopleinotherlegalcasescannotreasonablybeusedasanargumenttoestablishthataparticulardefendanthasbrokenthelaw.Suchevidencemightbeadmissibletodemonstrate,inunusualcases,thatpeopleareindeedsometimescapableofsurprisingbehaviour(forexample,amotherkillingherownchildren).Butaliberalsocietywouldnotbecontent,Ithink,withlegalpracticebasedonautilitarianprinciplethatifcrimeisrifethenthelawshouldconvictwithlowerstandardsofevidence,soastosuppresscrimeatthecostofimprisoningtheinnocent(Gardner-Medwin2005).Thiswouldbethe(p.475) actionofatotalitarianstate.Anenlightenedjurymustbepreparedtoacquitiftheevidencecouldplausiblyhavearisenwithoutguilt,howeverlikelyitmayseemthatthedefendantisguiltyonstatisticalgrounds.Ifcommoncrimesgounpunishedasaresult,thisneedstoberectifiedbyimprovingthequalityofevidenceratherthanbyloweringthethresholdforconviction.
UncertaintyintrialswithevidenceofadefinitecrimeTrialsareofcourseusuallymorecomplexandlessamenabletoquantitativeanalysisthanthecaricaturediscussedabove.Commonlythereisclearevidenceofacrime,andamongstthemanyhypothesesthatmightaccountforthisevidencetherearetwoofprimaryconcern:thoseputforwardbytheprosecutionanddefence.Theprosecutioncaseentailsguiltonthepartofthedefendant,whilethatofthedefencemaynotidentifyatallwhoisguilty,merelyofferingalternativeexplanationforevidencebroughtagainstthedefendantandaddingfurtherevidencethatmaytendtoproveinnocence.Boththeprosecutionanddefencehypothesesaretypicallycompositeinthesensethatthereareunknownelementsforwhichtheremayatbestbeareasonablesetofassignableprobabilities(forexamplethatthevictimeitherwalkedorgotaliftfromAtoB).Thesituationissomewhatsimilartocomparisonoftwoscientifichypotheses,wheretherelativelikelihoodofthedataarisingthroughthealternativeexplanationscanprovidearationalbasisforincreasingbeliefinoneortheother.However,aswesawabove,therecanbefundamentalproblemsinthisprocess.
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 10 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
Thefirstproblemistheneedforpriorprobabilities,fromwhichtogenerateposteriorprobabilitiesinthelightoftheevidence.Withscientifictheorieswesawhowtheremaysimplybenowayofresolvingdifferencesofopinionaboutsuchpriors.Similarly,incourtthepriorsmaydependonsuchdubiouslysubjectivefactorsasajurorsexpectationonbeingsummonedtotakepart,suppositionsabouttherigouroftheprosecutionservices,orhowthedefendantisdressedincourt.Aspectsofthebackgroundhistoryofthedefendant,whichmayormaynotbeallowedtoemergeincourt,mayreasonablybeconsideredrelevanttopriorprobabilitiesandtherebytotheposteriorprobabilityofguilt,butconstraintsanddebateoftencentreroundthefairnessofadmittingsuchevidenceandthelegalbalancebetweenpotentialprejudicialandprobativevalue(seeforexampleRobertsandZuckerman2004:chaps.4,11).
Asecondproblemabouttheuseofposteriorprobabilitiesarisesfromthecompositeandincompletenatureofthehypothesesunderconsideration.(p.476) Thedefencehypothesisseldomspecifieswhoactuallycommittedthecrime.Anyonefamiliarwithdetectivefiction(andprobably,thoughIdonothaveexperience,detectivefact)knowsthataprosecutioncasecanlookoverwhelminglystrongerthanadefencecaseuptothepointwhenanewideaoremergingfactsuddenlymakesplausibleahithertounconsideredsuspect,motiveoropportunity.Inanidealworldsuchadevelopmentwouldhavearisenandbeeninvestigatedbeforetrial,butmiscarriagesofjusticecanoccurifthisdoesnothappen.Thisisanalogoustothesimplecointossingexampleabove,wherethesameevidencewasseeneithertosupportornegatethehypothesisthatacoinisfair,dependingondetailsofthealternativehypothesesthatmayhavebeenthoughtof,andwhetherthesewereassignedsignificantpriorprobabilities.Miscarriagesofjusticecanveryeasilyoccurifthetruecriminalisabsentfromconsiderationorseemsabovesuspicion.
Theseproblemsconcerningprobabilityofguiltareanalogoustotheproblemsthatmakescientistsshyfromdebateabouttheprobabilitythataparticularhypothesisistrue.Theprobabilityofguiltcannothoweverbeavoidedaltogether,becauseitwouldobviouslybeunreasonabletoconvictsomeonewithoutsomehowconcludingthatthereisahighprobabilityofguilt.Butwehaveseenbothethicalconcernsandlogicalproblemsabouttreatingthisasawell-foundedsufficientcriterionforconviction.Hence,asinscience,itisalsoconstructivetofocusonthemorelimitedbutinsomerespectsmoredirectinferencesthatmaybedrawnfromdataprobabilities.Incourt,arelevantprobabilityiswhetheraninnocentpersoncouldhavebeenbroughttocourttofaceatleasttheweightofincriminatingevidencethathasbeenseenforthedefendant.Thiscanbeasubstantialconcernwherethereisahighprofilecrime,intensepoliceeffortandtheopportunitytotrawlawidepopulation:itbecomesquitepossiblethataninnocentsuspectmaybefoundagainstwhomaconvincingcasecanbemade.ThemurderinLondonofJillDandoandthesubsequentconvictionandacquittalofBarryGeorgecometomind.Ajurymustbepreparedtorecognisethataseeminglystrongcasecanoftenbemadeagainstapersoninnocentofthecrimeatissue.
Incourtthereisnotsuchacleardistinctiontobemadebetweenhypothesisanddata
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 11 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
probabilitiesasthereisinscience.Thisisbecausehypothesesandexplanationsofdataincourtarenotthewelldefinedstochasticmodelsthataretheusualbasisforcalculationofdataprobabilitiesinscience.Therearesubjectiveelementstotheprobabilitythatevidencemightarisewithoutguilt,forexamplehowlikelyitisthataparticularscenariowoulddeveloporthatwitnessesmaylie.Uncertaintyaboutwhetherevidencemightariseforaninnocentdefendantisthereforeinsomerespectsjustassubjectiveandarguableasisuncertaintyaboutthehypothesisofguilt.However,itinvolvesfewer(p.477) priorsandcanbejudgedwithinamorelimitedframework.Thisframeworkfocuseslargelyonthethingsthatcouldhappentoinnocentpeople,ratherthanonthebehaviourofcriminals.Anincidentaladvantage(thoughhardlyareasoninitselfforpreferringthisapproach)isthatsuchscenariosmaybeeasierforajurytoenvisage.
Selectionofsuspects:theDNAdatabasecontroversyAhazardfamiliartostatisticiansisthepotentialuseofdatatwiceover,oncetoselectahypothesisofinterest(inthiscase,topickasuspectfromapopulation)andthenagainasevidencethatthehypothesis(guiltofthissuspect)istrue.Therecanbeanelementofthisinpoliceprotocols(asintheSIDS/murdercases,probablytheBarryGeorgecaseandmanycasesinvolvingDNAmatching).Theprocedurestoavoidlogicalerrorsduetodoubleuseofdataarenotalwaysappreciatedbyscientists,soitisunsurprisingthatjuriesmayhavedifficultyhandlingtheissuecorrectly.Theymayevennotbeawarethatasuspectcametoattentionthroughatrawlofpolicerecordsratherthanthroughaconnectionwiththecase,sincethefactthatasuspecthadapolicerecordmaybeconsideredprejudicialincourt(seee.g.Kaye2009forexamplesandcourtrulingsinrelationtoDNAtesting).Therearediverseviewsandheateddebateinlegalandstatisticalcirclesabouthowtohandleevidencethatservesthesedualroles.Forexample,theAmericanNationalResearchCouncilCommitteeonDNAForensicScience(1996),Stockmarr(1999)andDevlin(2000)advocatedowngradingtheweightofevidencewhenthesuspectisidentifiedbyasinglematchinadatabasetrawl,whileBalding(2002,2005),Dawid(2002),Kaye(2009)andothersclaimthattheuseofadatabasedoesnotdiminishtheevidenceindeedstrengthensit,albeitusuallyonlyslightly,byrulingoutthoseinthedatabasewhotestnegative.Thedifferencescancorrespondtolargefactorsintheestimatedprobabilityofafalseconviction:factorscomparable(asshownbelow)withthenumberofpotentialsuspects.AsDawid(2002)pointsout,differencesbetweenstatisticalapproachestouncertaintymustrarelyhavesuchseriouspotentialconsequencesforthoseaffected.
Theissuescanbesomewhatclarifiedbyconsideringthesafetyofconvictionshowlikelytheproceduresaretoincriminateinnocentpersons.SupposethatthepersonwholeftDNAatacrimesceneisconsideredcertaintobethetrueperpetratorofacrime(TP),andthatthetechnicalprobabilityofthesamplematchingarandompersonotherthantheTPisaverysmallnumber(p),withnopossibilitythatamatchwouldfailtobeevidentiftheTPistested.Supposethereisaprimesuspectwhoisestimatedonthebasisofnongenetic(p.478) evidencetohavea50percentprobabilityofbeingguilty(s=50%:strongsuspicion,butcertainlynotenoughonitsowntoconvict).IftheDNAprofileofthissuspectistestedandmatchesthecrimesample,thentheprobabilitythatthismatchis
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 12 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
falseisp/(1+p),oralmostexactlyp.Convictiononthisbasishasasmallestimatedprobabilitypofbeingamiscarriageofjustice.Theriskisgreateriftheevidenceagainsttheprimesuspectisweaker(s0.5),thoughwithonlyasmallnumber(M)ofpossiblesuspectsthiswouldmakelittledifferenceforatestthatconfirmsaprimesuspect.2
CasesinvolvingtrawlofaDNAdatabasearemorecomplex.Supposethereareinitiallynoprimesuspectsbasedonnon-geneticevidence,butalargenumberNofpotentialsuspects(perhapsallthemalesofaplausibleagewithinacityonagivenday).SupposeaDNAdatabasecontainsprofilesforarandomsubsetincludingDofthesepossiblesuspects,andthatasearchofthedatabasehasrevealedexactlyoneoftheseasmatchingtheDNAatthecrimescene.ThereisattheoutsetaprobabilityD/Nthatthedatabaseincludesthetrueperpetrator(TP),andthereforeaprobability(D/N)(1p)D1thathewouldgiverisetoauniquematch,takingaccountoftheprobability(1p)D1thatnoadditionalmatcharisesbychanceforanyofthe(D1)innocentpersonsinthedatabase.Butthereisalsoaprobability(1D/N)pD(1p)D1thattheTPisnotinthedatabaseandthatexactlyoneoftheinnocentpersonsinthedatabasedoesmatch.Giventhefactthatauniquematchhasbeenfound,theprobabilitythatthisistheTPis1/(1+p(ND)).Thereisacomplementaryprobability(approximatelyp(ND)ifthisis1)thatthisisafalsematch,typicallymuchgreaterthantheprobabilityofafalsematchwhenasingletestiscarriedoutonaprimesuspect.3Thefactorbywhichthisriskisincreasedcomparedwiththeillustrativeexampleinthelastparagraphis(ND),thenumberofpotentialsuspectsoutsidethedatabasepotentially100,000ormoreinsomecases(Kaye2009).4OfcourseamatchfoundinaDNAtrawl(p.479) leadstoinvestigationofthesuspectandfurthernon-geneticevidence.Insomecasesthismayleadtoconfidentconviction(e.g.ifthesuspectturnsouttomatchaCCTVimagefromthecrimescene)oreliminationoracquittalifthematchisclearlyfalse(e.g.ifthereisadefinitealibi).ButDNAevidenceoftenappearstothejurytobethestrongestevidenceinacaseandconvictionfollowingaDNAtrawlmayevenresultwhenallotherevidenceseemstoweighinthedefendantsfavour(Donnelly2005;Kaye2009).ItiscrucialinsuchcasesthatajuryshouldrecognisethattheDNAdatacomesfromatrawlandcarriesamuchgreaterriskoffalseincriminationthanifitwereconfirmingsuspicionofaprimesuspect.Withoutthis,suchtrialsmustbeconsideredunsafe.
HowarewetoreconcilethisconclusionwiththeseeminglycontraryBayesianargumentpresentedbyDawid(2002)andKaye(2009)thattheorderinwhichDNAandnon-geneticdataareobtainedisimmaterial.Theyarguethatsinceatrawleliminatessuspectsaswellasidentifyingonewhomatches,whencombinedwithotherevidenceitwillrationallyleadtogreaterprobabilityofguiltthanwouldbeinferredifthesamepersonhadbeentestedastheprimesuspect.ThisargumentisillustratedinFigure17.1forthesimplecaseconsideredabove,wherethereisstrongnon-geneticevidence(E)againstadefendant,eitheremergingattheoutsettojustifyconfirmatoryDNAtestingofthedefendantasaprimesuspectorelseemergingafterthesuspecthasbeenidentifiedinatrawl.Thisshowshowtheoddsonthesuspectsguiltareaccumulatedtosimilaroutcomes,simplyinadifferentorder.Thisnear(p.480) equivalenceofoutcomeshasbeenusedtochallengethenotionthatjuriesneedtobeinformedofthespecialfactsand
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 13 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
statisticalconsiderationswhensuspectidentificationoccursthroughtrawlsormultipletesting(Balding2002,2005;Kaye2009).
Figure17.1 .ComparisonofBayesiantreatmentsofprobabilityofguilt.Arrowsshowsuccessivechangesinprobabilityofguiltandassociatedodds,forasuspectidentified(1)bynon-geneticevidence(E)followedbyaconfirmatoryDNAtest,or(2)throughtrawlinaDNAdatabasefollowedbyinvestigationanddiscoveryofE.Thenon-geneticevidenceonitsownleadstoaprobabilityofguilt(s)thatistakenheretobe50percent.N,Dandpareexplainedinthetext.Dashedarrowsshowthe(approximate)probabilityofaDNAmatchbeingfalsewitheachprocedure.Ifsuchamatchwereknowntobefalsethiswouldlead,givenassumptionsinthetext,tocertainacquittal.
Iwouldwhollyacceptthisargumentifthecollectionandevaluationofnon-geneticevidencewererigorous,quantitative,freeofselectionbiasanduncertainties,andderivedfrominvestigationofallpossibleavenues.Butthisisunrealistic,sometimesevenforevidencebackedbyscience.Therealityisthatmuchevidencedependsonhunchesandqualitativeargument,andthoroughinvestigationhasoftenbeenrestrictedtojustoneorafewofthepossiblesuspects.Thedegreeofsuspicionappropriateforaprimesuspectisoftenhighlyuncertain,andthediscoveryofapositiveDNAmatchprovidesstrongvindicationofthepotentiallyuncertainargumentsthatledtothestatusasprimesuspect.WhenasuspectisinsteadidentifiedbyDNAtrawl,theargumentsconstructedaroundnon-geneticevidenceobtainedafteridentificationwilllackthattestofauthenticityandmustretainalltheircaveatsanduncertainties.Contrastforexample,twosituations.5In(A),SbecomesasuspectbecauseanunreliableinformantassertsSdidit,SisDNAtestedandprovestomatchthecrimescene.In(B),SissuspectedbecausetrawlofadatabaserevealsthatheisaDNAmatchandthentheinformantaddshisassertionSdidit.Inneithercasewilltheinformantsblandassertioncarrysignificantweight,butitisessentialthatajuryshouldknowwhichscenarioappliestothecase,becausetheprobabilitythattheprocedurewouldproduceaDNAmatchincriminatinganinnocentpersonismuchgreaterin(B)than(A),byafactorequaltothenumberofplausiblesuspectswithinthedatabase.Aworthwhileapproachisforajurytoconsiderintrawlcaseswhether,ifsubsequentnon-geneticevidencehadbeenestablishedattheoutset,itwouldhavejustifiedthedefendantbeingtestedastheprimesuspect.OnlythenistheprobabilityoffalseincriminationbyDNAaslowastherandommatchprobabilityp.Incase(B)abovethisconsiderationwouldhighlightthecrucialimportanceofknowingwhethertheinformantspost-trawlassertionwasmadewithorwithoutknowledgethatSwasbythenasuspect.
Mythesishereisthatcourtsneedtoaddresstheprobabilitythatpoliceprocedures
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 14 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
(includingtrawlsforsuspects)couldhaveproducedevidenceagainstaninnocentpersonthatisatleastasincriminatingasthatpresented.Thisisaformofdataprobabilityandishowthecourtshouldultimatelyjudgethesafetyofaconviction.Itisajudgmentthattakesaccountofmorethanjustfactslinkingthedefendanttothecase,butsoitshould.Thedefendant(p.481) isincourtduetooperationoftheseprocedures,whichmustthereforebeunderscrutinyjustasmuchasthedefendanthimself.Ajudgmentthatfocusesonlyontheprobabilitythatthisdefendantisguiltywillnecessarilybedefi-cient.Itwouldessentiallyapportionprobabilitybetweenallpossiblesuspects,onlyoneofwhomisfullyinvestigatedandpresented,whilethetrueculpritandnatureofthecrimemaynotevenhaveenteredconsideration.Thereisastronganalogyheretothedifficultyofjudginghowlikelyitisthataparticularscientifichypothesisiscorrect:thisisalwaysuncertainbecausewesimplyneverknowifwehavehadtheluck,insightorimaginationtoconsiderthecorrectexplanationofthedataweobserve.
ConclusionAcknowledgementandcharacterisationofuncertaintiesarekeyelementsintheapplicationofknowledgeandevidence.Alackofawarenessofthenatureorextentofuncertaintycanleadtoconfusionandinappropriatedecisionsinareasofeducation,scienceandlaw.Myfocusherehasbeenonadistinctionthatcutsacrossthemajordebatesthatoftenemergeinstatistics(aboutthedefinitionofprobabilityinfrequentistorBayesianterms)andinlaw(aboutthemeritsofquantitativeorqualitativeapproachestodecisionmakingincourt).ThedistinctionIamconcernedwithisbetweenprobabilitiesassignedtohypothesesandprobabilitiesfortheobservationofdata,conditionalonaspecifichypothesis.Thereisnotachoicebetweenaddressingjustoneortheother:eachhasitsownlogicanditsdomainsofrigorousapplicationinbothscienceandlaw.Botharecentraltotheprocessofestablishingknowledge,whichitselfconsistsofgradationsofbeliefinhypothesesjustified,atleastpartially,byevidencefromdata.Thelinkbetweenhypothesisanddataprobabilitiesisseldomrigorousineitherscienceorlaw.Therearegreyareaswhereconclusionsmaybemattersofopinion.Scienceingeneralgetsroundthisproblembyrelyingontheindefiniteaccumulationofevidencethatwillultimatelyswampallbutthemostextremedifferencesofopinion.Criminallawreliesmoreonthemoderatingeffectofajurydecisiontogenerateanoutcomethatisgenerallyacceptedasreasonable.Inthelegalcontexthowever,Iarguethattheusualfocusonprobabilitythatthedefendantisguilty(ahypothesisprobability)isinadequate.Clearbenefitsariseifaverdictisconsideredtobeultimatelyconstrainedbyadataprobability:howlikelyisitthatsuchincriminatingevidencecouldhavearisenforaninnocentperson?
(p.482) Note.IamgratefultoD.KayeandD.Baldingandtotheeditorsofthisvolumeforconstructivecomments.
References
Bibliographyreferences:
Anderson,T.,Schum,D.andTwining,W.(2005),AnalysisofEvidence,2ndedn.(Cambridge,CambridgeUniversityPress).
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 15 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
Balding,D.J.(2002),TheDNAdatabasesearchcontroversy,Biometrics,58:2414.
Balding,D.J.(2005),Weight-of-EvidenceforForensicDNAProfiles(Chichester,JohnWiley).
Cohen,L.J.(1977),TheProbableandtheProvable(Oxford,ClarendonPress).
Dawid,A.P.(1986),ProbabilityForecasting,inEncyclopediaofStatisticalSciences,vol.7,ed.S.Kotz,N.L.JohnsonandC.B.Read(NewYork,Wiley-Interscience),pp.21018.
Dawid,A.P.(2002,)BayessTheoremandweighingevidencebyjuries,inBayessTheorem,ed.R.Swinburne,Proc.Brit.Acad.113:7190
Devlin,B.(2000),TheevidentiaryvalueofaDNAdatabasesearch,Biometrics,56:1276
Donnelly,P.(2005),Appealingstatistics,Significance,2:468.
Fisher,R.A.(1925),StatisticalMethodsforResearchWorkers(Edinburgh,OliverandBoyd).
Gardner-Medwin,A.R.(1995),Confidenceassessmentintheteachingofbasicscience,AssociationforLearningTechnologyJournal,3:805.
Gardner-Medwin,A.R.(2005),Whatprobabilityshouldajuryaddress?,Significance,2:912.Availableonlineat.
Gardner-Medwin,A.R.(2006),Confidence-basedmarkingtowardsdeeperlearningandbetterexams,inInnovativeAssessmentinHigherEducation,ed.C.BryanandK.Clegg(London,Routledge,TaylorandFrancisGroup).
Kaye,D.(2009),Roundinguptheusualsuspects:alegalandlogicalanalysisofDNAdatabasetrawls,NorthCarolinaLawReview,87:425503.
Kempthorne,O.andFolks,L.(1971),Probability,StatisticsandDataAnalysis(Ames,IA,IowaStateUniversityPress).
Lindley,D.V.(1972),BayesianStatistics:areview(Philadelphia,PA,SocietyforIndustrialandAppliedMathematics).
MacKay,D.J.C.(2003),InformationTheory,InferenceandLearningAlgorithms(Cambridge,CambridgeUniversityPress).
NationalResearchCouncilCommitteeonDNAForensicScience(1996),AnUpdate:theevaluationofDNAforensicDNAevidence(Washington,DC,NationalAcademyPress).
Popper,K.(1959),TheLogicofScientificDiscovery(London,Hutchinson).
Roberts,P.andZuckerman,A.(2004),CriminalEvidence(Oxford,OxfordUniversity
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 16 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
Press).
Royall,R.M.(1997),StatisticalEvidence:alikelihoodparadigm(London,ChapmanandHall).
RoyalStatisticalSociety(2002),LetterfromthePresidenttotheLordChancellorregardingtheuseofstatisticalevidenceincourtcases..
Senn,S.(2003),DicingwithDeath:chance,riskandhealth(Cambridge,CambridgeUniversityPress).
Song,Y.S.,Patil,A.,Murphy,E.E.andMontgomery,S.(2009),AverageprobabilitythataColdHitinaDNAdatabasesearchresultsinanerroneousattribution,J.Forensic.Sci.,54:227.
Stockmarr,A.(1999),LikelihoodratiosforevaluatingDNAvidencewhenthesuspectisfoundthroughadatabasesearch,Biometrics,55:6717.
Tillers,P.andGottfried,J.(2006),Acollateralattackonthelegalmaximthatproofbeyondareasonabledoubtisunquantifiable,Law,ProbabilityandRisk,5:13557.
vonNeumann,J.andMorgenstern,O.(1944),TheoryofGamesandEconomicBehavior(Princeton,NJ,PrincetonUniversityPress).
Notes:ProceedingsoftheBritishAcademy,171,465483.TheBritishAcademy2011.
(1)Forexample,measurementsthatarenormallydistributedrandomvariableswithmean,wherethepriorprobabilitydistributionforisessentiallyuniformoverarangemuchlargerthanthemeasurementstandarddeviation.
(2)Ifthepriorprobabilityofguiltofthematchingsuspectiss,basedonnon-geneticevidenceandeliminationofanypreviouslytestedsuspects,thentheprobabilitythatthismatchisfalseisapproximatelyp(1-s)/s(assumingthisitselfissmall).Evenifsfortheprimesuspectisconsideredunquantifiable,butthereareatmostMsuspectswhoareatallplausible,smustrationallybetakenasatleast1/Monthebasisthathisprobabilityofguiltmustbeconsideredatleastashighaseachoftheothers.Thissetsanupperboundontheprobabilityofafalsematchasp(M1).
(3)AdifferentderivationofthesameconclusionisgivenbySongetal.(2009).
(4)Notethatalargerandmorerelevantdatabase(largerD)reducestheriskthatauniquematchwillbeafalsematch.ThiscanbeseenasanargumentinfavourofmaintaininglargeDNAdatabases.Largedatabaseswouldalsoincreasethefrequencywithwhichdatabaseusewouldgeneratesuspectsforinvestigation.However,theremightbelegitimateconcernsthatinappropriatehandlingincourtofthestatisticalandtechnicalissuessurroundingdatabaseusemightleadtomorefalseconvictions.
-
Reasonable Doubt: Uncertainty in Education, Science and Law
Page 17 of 17
PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015
(5)IamgratefultoD.Kayeforsuggestingconsiderationofthisscenario.
Accessbroughttoyouby: PontificiaUniversidadCatolicadelPeru(PUCP)