Reasonable Doubt: Uncertainty in Education, Science and Law

Page 1 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

UniversityPressScholarshipOnlineBritishAcademyScholarshipOnline

Evidence,InferenceandEnquiryPhilipDawid,WilliamTwining,andMimiVasilaki

Printpublicationdate:2011PrintISBN-13:9780197264843PublishedtoBritishAcademyScholarshipOnline:January2013DOI:10.5871/bacad/9780197264843.001.0001

ReasonableDoubt:UncertaintyinEducation,ScienceandLawTONYGARDNER-MEDWIN

DOI:10.5871/bacad/9780197264843.003.0017

AbstractandKeywords

Thischapterchampionstheelicitationanduseofprobabilisticmeasuresofuncertainty.Itcomparestwodifferentviews(roughlycorresponding,respectively,totheBayesianandfrequentistapproachestostatisticalinference)astohowprobabilitiesshouldbeusedasevidence.Itarguesthat,whiletheformermaybealogicalideal,thelattermaybemoreappropriateinthespecialcircumstancesofthecourtroom.

Keywords:uncertainty,education,probabilities,evidence,statisticalinference

AbstractTheuseofevidencetoresolveuncertaintiesiskeytomanyendeavours,mostconspicuouslyscienceandlaw.Despitethis,thelogicofuncertaintyisseldomtaughtexplicitly,andoftenseemsmisunderstood.Traditionaleducationalpracticeevenfailstoencouragestudentstoidentifyuncertaintywhentheyexpress

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 2 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

knowledge,thoughmarkschemesthatrewardtheidentificationofreliableanduncertainresponseshavelongbeenshowntoencouragemoreinsightfulunderstanding.Inourinformation-richsocietytheabilitytoidentifyuncertaintyisoftenmoreimportantthanthepossessionofknowledgeitself.

Inbothscienceandlawtherearefundamentallydifferentkindsofuncertaintyatissue.Thereisuncertaintywhetheraparticularhypothesisiscorrect,andthereisuncertaintyaboutobservabledatathatmaybegeneratedifaparticularhypothesisiscorrect.Bothareexpressedintermsofprobabilities.Eachhasitsowndomainofapplicationanditsownlogic,buttheinterrelationshipiscomplexandsometimesmisunderstood.Hypothesisprobabilitiesarealwaysopentoerrorthroughpossiblefailuretotakeaccountofrealisticalternatives,whiletheproperinferencesthatcanbedrawnfromdataprobabilities(ofteninthecontextofsignificancetesting)arequitelimitedandeasilyover-interpreted.

Whenconsideringthesetwokindsofprobabilityinacourtoflawitispossibletointerpretthephrasereasonabledoubtindifferentways.Itcanbeseenasaddressingdatauncertainty:whethersuchincriminatingevidencemightwithreasonableprobabilityarisetoconfrontaninnocentperson.Or(themoreconventionalview)itcanbeseenassomesortofthresholdlevelontheprobabilitythatthedefendantisguilty(ahypothesisprobability).Eachtypicallyinvolveselementsofsubjectivejudgement,butfewerissuesanduncertaintiesarisewhenconsideringthedataprobabilityanditisarguedthatthisisoftenthemorecriticalandproperissueforajurytoaddress.ThishasparticularrepercussionsforcasesinvolvingidentificationofasuspectthroughtrawlofaDNAorotherdatabase.

(p.466) UncertaintyandmisconceptionineducationalassessmentASAUNIVERSITYTEACHER(inmedicalscience)Ihavetriedtoraisestudentsawarenessofuncertaintiesintheirownknowledge.Inconventionaleducationalassessmentstudentsareoftenmotivatedtohideuncertainties.Thisisperverse,becauseitisobviousthatitisagoodthingtobeabletodistinguishbetweenreliableandunreliableaspectsofonesknowledge.Studentsshouldinfactberewardedforacknowledginguncertaintiesthattheyhave.Byrewardingthem,forexamplewithsimplebutcarefullydesignedmarkschemes,theycanbeencouragedtoreflectonthenatureofanydoubts,andontheevidencethattheymaybeabletobringtobeartoresolvethesedoubts.Insteadofdoingthis,conventionalmarkingusuallyencouragesstudentstoblufftheirwaythrough:ittreatsluckyguessesinthesamewayaswelljustifiedknowledge,andfirmmisconceptionsasnoworsethanacknowledgedignorance.Theresultisago-for-itcultureinwhichdecisionsmaybetakeninthelightofamarginalorsuperficialpreferenceforsomeoption,withlittlefurtherthought.

Thedangersofsuchbehaviourareveryobviousinafieldsuchasmedicine,wherelivescanbelostthroughreluctancetoacknowledgeuncertainty.Eveninordinarydiscoursehowever,itisasignofweaknesstoshyfromproperawarenessandacknowledgementofonesuncertainties.Itisdisturbing,whentalkingtostudentsfamiliarwithessaywritingwhereyoumightthinkthediscussionofuncertaintieswouldbeparamountthatthey

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 3 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

oftensaytheywouldnotnormallymentiondoubtsaboutafactorargumenttheyaretemptedtoinclude.Anyonewhohasmarkedexamessaysknowsthatinpracticeaqualification,perhapsaquestionmarkinthemargin,willmakelittledifferencetoonesjudgementifthepointmadeiscorrect,whileitmakesabigdifferenceifuncertaintyisacknowledgedaboutsomethingincorrect.

Itmayatfirstseemparadoxicaltorewardsomeoneforacknowledginguncertainty.PerhapsitrunscountertosocietysmodernguidelinesforHowtogetoninpolitics,business,andmaybesometimesevenscienceorlaw.Buttherewardsforcautioninthefaceofuncertaintyarefundamentaltobiologicalsurvival.Childrenandyounganimalslearntheselessonsoftenthroughgames,anditisnocoincidencethattherelevantmathematicscomesundertheheadingGameTheory(vonNeumannandMorgenstern1944).Of(p.467) coursecertaintyalsobringsitsownrewardsjustaslongasitiscertaintyaboutsomethingthatiscorrect.Topursueacricketinganalogy,thedecisiontotrytohitaballforsixisfine,solongasoneisprettysureonecansucceed:thepriceoffailuretendstobehigh.Ifoneisuncertainofsuccess,thenamoremodeststrokewillonaveragebebetterrewarded.Exactlythesameprincipleisemployedinthecertainty-basedmarkschemeweemployforself-testsatUCLandImperialCollegeinLondon(Gardner-Medwin1995,2006).Acknowledginguncertainty(67%estimatedprobabilityofbeingcorrect)gives1markforacorrectanswerand0ifincorrect.Claimingahighlevelofcertainty(80%)gives3marksor6,whileanintermediatelevelgives2marksor2.Thisisaproperormotivatingmarkscheme,inthesensethatastudentalwaysexpectstogainbygivinganaccurateindicationofhowreliablehe/shethinksthechosenansweris(Dawid1986).Withthisschemethereisnobenefittobegainedbythestudenttryingtoplaythesystem:pretendingtobeeithermoreorlessconfidentthanhe/shereallyis.Itisremarkablehowreadilystudentstaketothisscheme,appreciatingthegoodsenseandutilitarianvalueofsettlingformodestrewardsandpenaltieswhenunsure.

Uncertaintyabouttheexpressionofknowledgeorunderstandingisanexampleofuncertaintyaboutideas.Muchofeducationaltestingrevolvesaroundideasthataregenerallyaccepted(atleastwithinanagreedframework)asdefinitelyeithertrueorfalse,thoughstudentswithpartialknowledgemayassignprobabilitieslessthan1andthosewithseriousmisconceptionsmayassignverylowprobabilitiestowhatiscorrect.Thereareofcoursemanyelementstoeducationalexperiencewherethereisnocertaintyandthetaskistogenerateordiscussideasthatareneitherknownnoragreedtobeobjectivelycorrectorincorrect.Herewecomeclosertotheissuesthatariseinscientificresearchandincourtsoflaw.

Uncertaintyaboutideas:thedrivingforceinscienceScientistsworkwithtwofundamentallydifferentkindsofuncertainty,thoughtheydontalwaysdistinguishthemveryclearly.Thefirstisanintrinsicallysubjectiveprobability:Howcertainisitthataparticularideaorhypothesisiscorrect?Thisisobviouslyfundamentalwithinscience,buttheanswerstosuchquestionsarealwayssubjectiveinthesensethattheymaydifferevenbetweenwellinformedandintelligentindividuals.Bygainingadditionalevidenceonemayshiftsuchsubjectiveprobabilitiesinwaysthatall

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 4 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

agreearerationalandoccasionallyevenquantifiable.Indeed,asPopper(1959)pointedout,onecansometimesdemonstratethatanideaiscertainlywrong,thoughneverthatitis(p.468) certainlyrightasageneraltruth.Differentlevelsofscepticismanduncertaintyamongstindividuals,aboutwhatiscorrect,mayneverbeeliminated.Nevertheless,thereareofcoursegreatswathesofscientificconclusionsthatareregarded,bymoreorlessallwhohavestudiedthem,asessentiallycertain.Itisthepowerofsciencethattheaccumulationofevidence,givenenoughtimeandeffort,tendstobecomeoverwhelmingononesideortheotherofanyargument.However,evidencedoesnotalwaysdiminishuncertainty,andevenfirmlyembeddedideascanoccasionallybechallengedoroverturned,evenbysimplepiecesofevidence.Uncertaintyaboutideasisthemotivationforprogressinscience,despiteitsnebulousandlargelyunquantifiablenature.Itdrivesthetestingofpredictionsandthedevisingofalternativetheoriesandexperimentalchallenges.

Darwinstheoryofnaturalselectionisaprimeexampleofanideathatwasconceivedwithahugeelementofuncertainty,notleastinDarwinsmind.Darwinsgreatsuccesswasthatheamassedavastamountofevidenceinitsfavourbeforegoingpublic:enoughevidencetodiminishhisownuncertaintyandtoshaketheconvictionofmanysceptics.Thereneverwas(andneverwillbe)atimewhenDarwinstheoryistotallycertain,beyondanymeaningfulthreshold.Ofcourse,nearlyeveryscientistbelieves,asIcertainlydo,inthistheory.Darwinsideaisasfirmlyembeddedasanyinscience,thankstotheprogressiveelucidationofbiologicalmechanismsandtheaccumulationofdatathatareall(sofarasIknow)consistentwiththecoreelementsandpredictionsofthistheory.Buttherecanbenocertaintythatnewideasandobservationscouldnotchangeit.AscientistsnaturalresponsetothequestionIsDarwinstheorybeyondreasonabledoubt?istosayWhatdoubt?Letsseeifwecandeviseatestforanyideasabouthowitmaybewrong.Scientistslovecoherentscepticism,justsolongastheideasaretestable.Thehistoryofscienceteachesthemtoregarddogmaticbeliefsaspointlessandarrogant.Thestrengthofthechieftenetsofscienceisnotthattheyarecertain,butthattooverturnthemwouldrequirenotonlyobservationsinconsistentwiththem,butalsoanewexplanationforalltheevidencethathashithertoappearedtosupportthem.Darwinachievedthisinrelationtothenotionofdivinecreation,asdidEinsteinwithclassicalmechanics.Scientistsseldomofcourseachievesuchdramaticrevolutionaryimpact;butevenwhenthepursuitofuncertaintydoesntoverturnorstrengthenexistingideas,itoftenthrowslightonfacetsthatwerepreviouslyunclear.

(p.469) UncertaintyaboutexpecteddataThesecondkindofuncertaintyisconceptuallymorestraightforwardandquantitative,thoughoftenunderpinnedbycomplexmathematics.Thisisuncertaintyaboutthedatathatwillariseinanexperiment,giventhattherelevantmechanismsandprinciplesareeitherfullyunderstoodorfullydefinedbyahypotheticalmodel.Statisticalanalysisofsuchuncertaintyispartofanyscientifictraining(andthebaneofmanystudents!).Itgivesaframeworkforevaluationofnewresults,sinceanovelclaimisoflittlevalueifthenoveltyorinterestoftheevidencecouldquitelikelyhavearisenwithconventionaloruninterestingassumptions.Thecommonestformatisthatofsignificancetesting,initiated

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 5 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

byFisher(1925).TheassumptionsonwhichasignificancetestisbasedaredescribedasanullhypothesisH0.Apointofpotentialinterestisencapsulatedinastatisticfromtheobservations,quantifyingadifferencefromthemeanormedianvalueexpectedonH0.Theoutcomeofthetestistheprobability(P-value)basedonH0that,withtheproceduresadoptedintheresearch,atleastaslargeavaluewouldbeobserved.

SincetheuseandmisuseofsignificancetestsandP-valueshasahugeandcontroversialliterature(seeforexampleveryapproachablereviewsbyRoyall(1997)andSenn(2003)),Ishallpresentherejustapersonalperspectiveonsomeoftheseissues.Afundamentalpoint,clearIhopeinthedescriptionabove,isthataP-valueisaprobabilityofobservingdata,conditionalonaspecifichypothesis.Itdoesnotexpressuncertaintyabouthypotheses.P-valuesarehugelyvaluableinsciencebecausetheyquantifythedegreetowhichdataareconsistentwithsomeconventionalorpostulatedidea.PerhapsthemostusefulroletheyperformiswhentheP-valueislargeandtheresultisnot-significant:thismeansthatthepointofinterestinthedatawouldbequitelikelytoariseevenifH0istrue,sothereislittlereasontopayattentiontothisaspectofthedataasanykindofchallengetoH0.Onecanconcludethisquitestraightforwardly,withoutanyconsiderationofalternativehypothesesortherathernebulousbusiness,discussedinthelastsection,ofaddressingtheprobabilitiesthathypothesesmaybecorrect.

AhazardintheinterpretationofP-valuesarisesfromterminologyoftenusedtoexpressconclusionsbasedonthem.LowP-valuesareoftensaidtojustifyrejectionofH0ataparticularsignificancelevel.Thisisonlytrueinaveryrestrictedformalsense,whenaP-valueisusedasadecisioncriterionbetweentwoactionsthatwouldbeappropriateifthereweresolidreasonstoacceptorrejectH0.ForexampleonemightdecidetoignorethedatabecauseitisconsistentwithH0,orstartresearchingalternativehypothesesifthedatawouldbesurprisingonthebasisofH0.Iftherearealternativehypotheses(p.470) alreadyformulated,withsomebasisforassigningthemrelativeprobabilities,thenaresultwithalowP-valuedoesntnecessarilyevenargueagainstH0,letalonejustifyrejectingit.AninterestingresultmaybesoextremethatitisveryunlikelyonH0,butitmaybeevenlesslikelyonwhateveralternativehypothesesareconsideredplausible.Asanexample,supposethattentossesofacoinyieldnineheadsandonetail.Doesthisresult,surprisinglyfarfrom50:50(P-value=2percent)argueforrejectionofahypothesis(H0)thatthisisafaircoin?Iftheonlyalternativesseemtobethatacoinmighthavetwoheadsortwotails,thenthedataclearlyshowsbothofthesefalse,supportingH0.However,ifoneenvisagesthepossibilitythatacoinmightbebiassedsoastobemorelikelytolandoneway,andconsidersthisplausible,thenthedatalendmoresupporttothisideathantoH0.WhetheroneendsupconsideringH0moreorlesslikelythanattheoutsetdependsontheinitialprobabilitiesoneassignstoanyofthese(andpossiblyother)ideasonespriorprobabilities.Inscience,especiallyatthefrontiers,thealternativestoawell-formulatedhypothesisareoftensimplyamatterofspeculation.InthatsituationallonecansensiblysayaboutdatawithalowP-valueisthattheparameterofinterestissurprisinglyfarfromexpectationonthebasisofH0(orsimilarhypotheses)andthatotherhypotheses,notnecessarilyveryplausibleoreventhoughtof,couldhaverendereditlesssurprising.

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 6 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

TheborderlinebetweenuncertaintyabouthypothesesanddataAusefulextensionofsignificancetestsisthecalculationofso-calledconfidencelimits:arangeofnullhypothesesthatwouldyieldP-valuesaboveaspecifiedlevelforaresultofinterestinthedatainotherwords,theywouldmaketheresultunsurprising.Thispracticehelpstoclarifyhowusefultheobservationsare:howwelltheyservetodiscriminatebetweendifferentvaluesofaparameterwithinahypotheticalmodel;butitstillsaysnothingabouttheprobabilitythatoneshouldassigntothecorrectnessofanyofthesemodels.Thecommonlyusedterminologyisagain,inmyview,misleadingbecausetheterm95percentconfidencelimitsforaparameterinamodelcaneasilybeunderstoodtomeanthatthereisa95percentprobabilitythatthetrueparameterlieswithintheselimits.Thisissimplynotso,asisevidentifyouthinkaboutanexperimentthattriestomeasureaneffectthatyouconsidertobealmostcertainlynon-existent(perhapsextrasensoryperceptionorhomeopathy).Whatevertheprotocolfortheexperiment,youshouldexpecton5percentofoccasionsthataresultwillbeobtainedforwhichanileffectisoutsidethecalculatedconfidencelimits.Ifyouhappentoexperienceoneof(p.471) theseresultsitwouldobviouslybeinappropriatetoadoptabeliefthatthereisa95percentprobabilitythatthepostulatedeffectisreal.Perhapsconsonancelimits(KempthorneandFolks1971)wouldbetterconveythetruemeaning,whichisarangeofmodelsforwhichthedatawouldberelativelycommonplacefallingwithinthe95percentmostlikelyoutcomes.

Ingeneralitisimportanttomakeacategorydistinctionbetweentheprobabilitythatdatawillariseandtheprobabilitythatahypothesisaboutthegenerationofdataiscorrect.Sometimes,however,thereisasimplerelationshipbetweenthetwo.Forexample,anexperimentmayestimate(bysamplemeasurements)themeanvalueofaparameterthatisconsideredattheoutsetequallylikelytotakeanyofawiderangeofvalues:perhapssomecharacteristicofanewlyencounteredsubstance,objectorspecies.Then(withafew,oftenjustifiable,assumptions)1theposteriorprobabilitiesofdifferenthypothesesaboutthetruemeanarethesameasthedataprobabilitiesthatwouldbecalculatedforsamplemeans,onthehypothesisthatthetruemeaniswhatwasactuallymeasured.Returningtoterminology,thismeansthatwhennohypothesisinitiallyhaspreferredstatus,thelimitscautiouslydescribedaboveas95%consonancelimitsaretruly95percentconfidencelimitstheyidentifyarangethatonecanbefairlyconfidentincludesthetruemean(with95percentprobabilitybasedononlythesedata).Exampleslikethiswhereonecanconvertdirectlyfromadataprobabilitytoahypothesisprobabilityarenotrare,buttheyleadtoconfusioniftheirlimitationsarenotproperlytakenintoaccount.

Whenalternativehypothesesarewelldefinedanddohavespecialstatusinanexperiment,forexamplerivalcausalexplanationsofthedata,thenupdatingofprobabilitiesforthealternativesrequiresBayesiananalysis(Lindley1972).Aswesawabove(inthecoinexample)attemptstomakesuchinferencesusingP-valuescanbequitemisleadingsinceanunlikelyresultonahypothesisH0mayactuallybeevidenceinfavourofH0.Bayesiananalysisinvolvesidentifyingpriorprobabilitiesforthevarioushypothesesandupdatingtheseinthelightofthedata,usinglikelihoodratios(therelativeprobabilitiesthatthedatawouldbeobservedonthedifferenthypotheses).Thetechnicalitiesneednot

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 7 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

concernushere,buttheprocedureisdemonstrablysuccessfulinsituationswherealternativehypotheseshaveclearpriorprobabilities.Evenwhenthisisnotthecase,andpriorprobabilitiesarelittlemorethanguesses,sufficientdatacansometimesleadtohighlyreliableinferences(seeforexample,(p.472) MacKay2003).Themainreservationisthatconclusionscanbewhollymisleadingifthecorrecthypothesishasbeenomittedfromconsiderationorassignedaninappropriateprobability.Ifthisisamajorrisk,asitcanofcoursebeatthefrontiersofresearch,thenthemorelimitedlogicbywhichsignificancetestschallengeindividualhypothesescanbeamorecomfortablebasisforscientificprogress.

AnimportantapplicationofBayesiananalysisisinmedicaldiagnosis.Hereonestartswithfairlyobjectiveprobabilitiesthatdifferenthypothesesmightbetrue,forexamplethatapersonwithapersistentcoughmayhaveeachofanumberofdifferentdiseases.Thentheseprobabilitiesareupdatedonthebasisofdatafromthepatientshistoryandtestscarriedout,usinginformationabouthowfrequentlythedatawouldariseinthepopulationforeachoftheconditions.Thisprocessmaysimplyoccurinthedoctorsmind,leadingeventuallyeithertoconfidenceaboutspecificdiagnosesortoresidualuncertainty.Alternatively,aBayesiancomputeralgorithmmayprocessthedatatogenerateexplicitposteriorprobabilities.Whichisbetterisdebatable,giventhepossibleunreliabilityofthedoctorscompilationofinferencessetagainsthisorherabilitytotakeintoaccountsubtleaspectsofthedataignoredbyacomputeralgorithm.Butthereisnodisputingtheappropriatenessofthisstrategytoresolvemedicaluncertaintiesandtohelpmakedecisions.

UncertaintyincriminaltrialsTheuncertaintiesinacriminaltrialhavesuperficialsimilaritiestotheproblemsofmedicaldiagnosis.Theendpointisadecisionthatturnsgradeduncertaintiesintospecificactions:whethertoconvictadefendant,orapplytreatmentforaspecificdiagnosis.Therelativeutilitiesofrightandwrongdecisionsmayinpartdeterminewhatlevelofdoubtisacceptableforafinaldecision.Inmedicinethisuncertaintyisnormallyexplicit,sincethepatientusuallymakesthefinaldecisionaboutwhethertoaccepttreatment,andmustthereforebeinformedofdoubtsandlikelyoutcomes.Inlaw,thecriterionforconviction(beyondreasonabledoubt)issubstantiallyopentoajurysinterpretation,withonlypartialknowledgeoftheutilitiesinvolvedsinceasentenceforconvictionmayonlybedeterminedatalaterstagebyajudge.Butthesearenotprofounddifferences.

Legalcasesoftenhavemanylayersofcomplexity,forexampleconcerningfacts,motives,identification,intentionandwitnesscredibility.Finaldecisionsmayhingeonacomplexsynthesisofmanytypesofdoubt(Cohen1977;Andersonetal.2005),anditisfarfromclearthatmathematicalapproaches(p.473) arehelpfulincomingtoappropriateconclusions.Alternativeconceptionsofprobability(forexampleCohensinductiveprobabilitybasedonqualitativeeliminativereasoning)maysometimesbemorerational(oratleastmoremanageableforajury)whendecidingwhetherfactsarebeyondreasonabledoubtinatrial.Suchissuesaremuchdebatedinthelegalliterature(seeforexampleTillersandGottfried2006)withcogentargumentsfrommanyperspectives.

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 8 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

However,thedebatethatfollowsnaturallyfromthediscussionhereonthenatureofuncertaintiesinscienceisnotaboutwhetherprobabilitiesshouldbequantifiedbutwhetherweshouldbedealingwithuncertaintiesabouthypothesesoraboutdata.Ideally,ofcourse,onewouldlikecourtdecisionstobebasedoncertaintyratherthanuncertainty,butthisisrarelythecase.Iftheevidencedoessupportcertainty,thenconcludingthatdefendantDdefinitelycommittedcrimeC(anassertionaboutahypothesisprobability)isequivalenttoconcludingthattheevidencecouldnothavearisenifDwereinnocent(anassertionaboutadataprobability):eachconclusionfollowsfromtheother.Butifuncertaintiesarepresent,thesebecomedifferentquestions.

Ihavearguedelsewhere(Gardner-Medwin2005)thatitcanberationalforajurytocometotheconclusionthatthehypothesisofguiltisveryprobablycorrect,whileacquittingonthegroundsthattheevidencecouldwithreasonablelikelihoodhavearisenforaninnocentperson.Itmayatfirstseemthattheseissuesareoppositesidesofthesamecoin,whichwouldmakemyassertioneitherparadoxicalormerelyamatterofsettingdifferentthresholdsforjudgingthetwoprobabilities.Iwillfirstsetoutacaricatureexample,basedoncasesthatfirstdrewmyinterest(asdiscussedbyDawid2002)toshowthatthisisnotthecase.

Supposemultipleinfantsinafamilyhavediedincircumstancesconsistentwitheithersuddeninfantdeathsyndrome(SIDS:araremedicalconditionthatleavesnospecificsignspostmortem)orinfanticidebythemother.Thismayreasonablyleadtosuspicionofcrime,andindeedcaseshavecometocourtwithlittlemoreinthewayofpertinentevidence.Itisobviouslyrelevanttoaskquestionsanalogoustothoseunderlyingtestsofsignificance:Howlikelyisitthatsuchevidencewouldariseinaninnocentfamilywithcomparablegenetic,medicalandsocio-economicbackground,andhowoftenwouldsuchcasesbeexpectedtoariseinaninnocentpopulation?Suchquestionscannotbeansweredwithprecision,butcompetentexpertsshouldbeabletogivereasonablerangesfortheanswers.UnfortunatelythesupposedlyexperttestimonyinrecentUKcaseswasnotcompetent,raisingseriousconcernsabouthowexperttestimonyshouldbevalidated(RoyalStatisticalSociety2002);butthisdoesnotdetractfromtherelevanceofproperanswers.Theconclusionmightbethattheriskofthedeathsthathavebroughtthismother(p.474) tocourtarisinginherfamilywithoutcrimewasatleastaprobabilityP,andthatsimilarmultipledeathsinfamilieswithnogreaterriskfactorsformultipleSIDSmightbeexpectedtoarisesomewhereintheUKonceeveryXmonths.Ajurymightreasonablydecidetoacquitsimplyonthebasisofsuchtestimony,justifyingtheirdecisionbysayingthatifjuriesconvictonsuchevidencetheymightberesponsibleforconvictinginnocentmothersattherateofoneeveryXmonthsintheUK,whichtheydeemunacceptable.Theacceptabilitycriterionisofcourseasubjectivematter:howmanyfalseconvictionsofthissortmightbeacceptableinagivenpopulationperyear,decadeorcenturyintheinterestsofjustice.Butitisnomoresubjectivethanwhatconstitutesreasonabledoubtonanyotherbasis.

Analternativeapproachistodecidesuchacasebasedontheprobabilitythatthehypothesisofguiltiscorrect,thatisthatthisdefendantisguiltywithinthemeaningofthe

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 9 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

law.Thisviewisoftenimplicitinthelegalliteratureandthemedia.ItwasclearlyexpressedintheletterfromtheRoyalStatisticalSociety(2002)totheLordChancellorintheUKconcerningthemultipleSIDScaseofRvs.SallyClark,andwasanalysedinrelationtothiscaseinmoredetailbyDawid(2002).Toquotetheletter:Twodeathsbymurdermaywellbeevenmoreunlikely[thantwodeathsbySIDS].Whatmattersistherelativelikelihoodofthedeathsundereachexplanation,notjusthowunlikelytheyareunderoneexplanation.Theimplication(madeexplicitbyDawid(2002)inSection2.3ofhispaper)isthatajuryshoulduseevidenceabouttheincidenceofthistypeofmurderinthepopulation,andconvictifthisissufficientlygreaterthanthatofSIDS,takingaccountofalltheknowncircumstances.Thisiscertainlyarationalwaytoinferprobabilityofguiltinlightoftheevidence,butithasuncomfortableconsequenceswhenusedasthebasisforconvictionoracquittal.Inparticular,itmeansthatadefendantwhowouldbeacquittedonthegroundsthattheevidenceisreasonablyconsistentwithinnocence(theargumentofthelastparagraph)mightfindherselfconvictedbecausesufficientlymanyotherpeoplehavecommittedthecrimeofwhichsheisaccused(Gardner-Medwin2005).Thisseemsethicallyimproperandprobablyunacceptableinlaw,becausethevoluntarycriminalactsandintentionsofpeopleinotherlegalcasescannotreasonablybeusedasanargumenttoestablishthataparticulardefendanthasbrokenthelaw.Suchevidencemightbeadmissibletodemonstrate,inunusualcases,thatpeopleareindeedsometimescapableofsurprisingbehaviour(forexample,amotherkillingherownchildren).Butaliberalsocietywouldnotbecontent,Ithink,withlegalpracticebasedonautilitarianprinciplethatifcrimeisrifethenthelawshouldconvictwithlowerstandardsofevidence,soastosuppresscrimeatthecostofimprisoningtheinnocent(Gardner-Medwin2005).Thiswouldbethe(p.475) actionofatotalitarianstate.Anenlightenedjurymustbepreparedtoacquitiftheevidencecouldplausiblyhavearisenwithoutguilt,howeverlikelyitmayseemthatthedefendantisguiltyonstatisticalgrounds.Ifcommoncrimesgounpunishedasaresult,thisneedstoberectifiedbyimprovingthequalityofevidenceratherthanbyloweringthethresholdforconviction.

UncertaintyintrialswithevidenceofadefinitecrimeTrialsareofcourseusuallymorecomplexandlessamenabletoquantitativeanalysisthanthecaricaturediscussedabove.Commonlythereisclearevidenceofacrime,andamongstthemanyhypothesesthatmightaccountforthisevidencetherearetwoofprimaryconcern:thoseputforwardbytheprosecutionanddefence.Theprosecutioncaseentailsguiltonthepartofthedefendant,whilethatofthedefencemaynotidentifyatallwhoisguilty,merelyofferingalternativeexplanationforevidencebroughtagainstthedefendantandaddingfurtherevidencethatmaytendtoproveinnocence.Boththeprosecutionanddefencehypothesesaretypicallycompositeinthesensethatthereareunknownelementsforwhichtheremayatbestbeareasonablesetofassignableprobabilities(forexamplethatthevictimeitherwalkedorgotaliftfromAtoB).Thesituationissomewhatsimilartocomparisonoftwoscientifichypotheses,wheretherelativelikelihoodofthedataarisingthroughthealternativeexplanationscanprovidearationalbasisforincreasingbeliefinoneortheother.However,aswesawabove,therecanbefundamentalproblemsinthisprocess.

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 10 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

Thefirstproblemistheneedforpriorprobabilities,fromwhichtogenerateposteriorprobabilitiesinthelightoftheevidence.Withscientifictheorieswesawhowtheremaysimplybenowayofresolvingdifferencesofopinionaboutsuchpriors.Similarly,incourtthepriorsmaydependonsuchdubiouslysubjectivefactorsasajurorsexpectationonbeingsummonedtotakepart,suppositionsabouttherigouroftheprosecutionservices,orhowthedefendantisdressedincourt.Aspectsofthebackgroundhistoryofthedefendant,whichmayormaynotbeallowedtoemergeincourt,mayreasonablybeconsideredrelevanttopriorprobabilitiesandtherebytotheposteriorprobabilityofguilt,butconstraintsanddebateoftencentreroundthefairnessofadmittingsuchevidenceandthelegalbalancebetweenpotentialprejudicialandprobativevalue(seeforexampleRobertsandZuckerman2004:chaps.4,11).

Asecondproblemabouttheuseofposteriorprobabilitiesarisesfromthecompositeandincompletenatureofthehypothesesunderconsideration.(p.476) Thedefencehypothesisseldomspecifieswhoactuallycommittedthecrime.Anyonefamiliarwithdetectivefiction(andprobably,thoughIdonothaveexperience,detectivefact)knowsthataprosecutioncasecanlookoverwhelminglystrongerthanadefencecaseuptothepointwhenanewideaoremergingfactsuddenlymakesplausibleahithertounconsideredsuspect,motiveoropportunity.Inanidealworldsuchadevelopmentwouldhavearisenandbeeninvestigatedbeforetrial,butmiscarriagesofjusticecanoccurifthisdoesnothappen.Thisisanalogoustothesimplecointossingexampleabove,wherethesameevidencewasseeneithertosupportornegatethehypothesisthatacoinisfair,dependingondetailsofthealternativehypothesesthatmayhavebeenthoughtof,andwhetherthesewereassignedsignificantpriorprobabilities.Miscarriagesofjusticecanveryeasilyoccurifthetruecriminalisabsentfromconsiderationorseemsabovesuspicion.

Theseproblemsconcerningprobabilityofguiltareanalogoustotheproblemsthatmakescientistsshyfromdebateabouttheprobabilitythataparticularhypothesisistrue.Theprobabilityofguiltcannothoweverbeavoidedaltogether,becauseitwouldobviouslybeunreasonabletoconvictsomeonewithoutsomehowconcludingthatthereisahighprobabilityofguilt.Butwehaveseenbothethicalconcernsandlogicalproblemsabouttreatingthisasawell-foundedsufficientcriterionforconviction.Hence,asinscience,itisalsoconstructivetofocusonthemorelimitedbutinsomerespectsmoredirectinferencesthatmaybedrawnfromdataprobabilities.Incourt,arelevantprobabilityiswhetheraninnocentpersoncouldhavebeenbroughttocourttofaceatleasttheweightofincriminatingevidencethathasbeenseenforthedefendant.Thiscanbeasubstantialconcernwherethereisahighprofilecrime,intensepoliceeffortandtheopportunitytotrawlawidepopulation:itbecomesquitepossiblethataninnocentsuspectmaybefoundagainstwhomaconvincingcasecanbemade.ThemurderinLondonofJillDandoandthesubsequentconvictionandacquittalofBarryGeorgecometomind.Ajurymustbepreparedtorecognisethataseeminglystrongcasecanoftenbemadeagainstapersoninnocentofthecrimeatissue.

Incourtthereisnotsuchacleardistinctiontobemadebetweenhypothesisanddata

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 11 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

probabilitiesasthereisinscience.Thisisbecausehypothesesandexplanationsofdataincourtarenotthewelldefinedstochasticmodelsthataretheusualbasisforcalculationofdataprobabilitiesinscience.Therearesubjectiveelementstotheprobabilitythatevidencemightarisewithoutguilt,forexamplehowlikelyitisthataparticularscenariowoulddeveloporthatwitnessesmaylie.Uncertaintyaboutwhetherevidencemightariseforaninnocentdefendantisthereforeinsomerespectsjustassubjectiveandarguableasisuncertaintyaboutthehypothesisofguilt.However,itinvolvesfewer(p.477) priorsandcanbejudgedwithinamorelimitedframework.Thisframeworkfocuseslargelyonthethingsthatcouldhappentoinnocentpeople,ratherthanonthebehaviourofcriminals.Anincidentaladvantage(thoughhardlyareasoninitselfforpreferringthisapproach)isthatsuchscenariosmaybeeasierforajurytoenvisage.

Selectionofsuspects:theDNAdatabasecontroversyAhazardfamiliartostatisticiansisthepotentialuseofdatatwiceover,oncetoselectahypothesisofinterest(inthiscase,topickasuspectfromapopulation)andthenagainasevidencethatthehypothesis(guiltofthissuspect)istrue.Therecanbeanelementofthisinpoliceprotocols(asintheSIDS/murdercases,probablytheBarryGeorgecaseandmanycasesinvolvingDNAmatching).Theprocedurestoavoidlogicalerrorsduetodoubleuseofdataarenotalwaysappreciatedbyscientists,soitisunsurprisingthatjuriesmayhavedifficultyhandlingtheissuecorrectly.Theymayevennotbeawarethatasuspectcametoattentionthroughatrawlofpolicerecordsratherthanthroughaconnectionwiththecase,sincethefactthatasuspecthadapolicerecordmaybeconsideredprejudicialincourt(seee.g.Kaye2009forexamplesandcourtrulingsinrelationtoDNAtesting).Therearediverseviewsandheateddebateinlegalandstatisticalcirclesabouthowtohandleevidencethatservesthesedualroles.Forexample,theAmericanNationalResearchCouncilCommitteeonDNAForensicScience(1996),Stockmarr(1999)andDevlin(2000)advocatedowngradingtheweightofevidencewhenthesuspectisidentifiedbyasinglematchinadatabasetrawl,whileBalding(2002,2005),Dawid(2002),Kaye(2009)andothersclaimthattheuseofadatabasedoesnotdiminishtheevidenceindeedstrengthensit,albeitusuallyonlyslightly,byrulingoutthoseinthedatabasewhotestnegative.Thedifferencescancorrespondtolargefactorsintheestimatedprobabilityofafalseconviction:factorscomparable(asshownbelow)withthenumberofpotentialsuspects.AsDawid(2002)pointsout,differencesbetweenstatisticalapproachestouncertaintymustrarelyhavesuchseriouspotentialconsequencesforthoseaffected.

Theissuescanbesomewhatclarifiedbyconsideringthesafetyofconvictionshowlikelytheproceduresaretoincriminateinnocentpersons.SupposethatthepersonwholeftDNAatacrimesceneisconsideredcertaintobethetrueperpetratorofacrime(TP),andthatthetechnicalprobabilityofthesamplematchingarandompersonotherthantheTPisaverysmallnumber(p),withnopossibilitythatamatchwouldfailtobeevidentiftheTPistested.Supposethereisaprimesuspectwhoisestimatedonthebasisofnongenetic(p.478) evidencetohavea50percentprobabilityofbeingguilty(s=50%:strongsuspicion,butcertainlynotenoughonitsowntoconvict).IftheDNAprofileofthissuspectistestedandmatchesthecrimesample,thentheprobabilitythatthismatchis

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 12 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

falseisp/(1+p),oralmostexactlyp.Convictiononthisbasishasasmallestimatedprobabilitypofbeingamiscarriageofjustice.Theriskisgreateriftheevidenceagainsttheprimesuspectisweaker(s0.5),thoughwithonlyasmallnumber(M)ofpossiblesuspectsthiswouldmakelittledifferenceforatestthatconfirmsaprimesuspect.2

CasesinvolvingtrawlofaDNAdatabasearemorecomplex.Supposethereareinitiallynoprimesuspectsbasedonnon-geneticevidence,butalargenumberNofpotentialsuspects(perhapsallthemalesofaplausibleagewithinacityonagivenday).SupposeaDNAdatabasecontainsprofilesforarandomsubsetincludingDofthesepossiblesuspects,andthatasearchofthedatabasehasrevealedexactlyoneoftheseasmatchingtheDNAatthecrimescene.ThereisattheoutsetaprobabilityD/Nthatthedatabaseincludesthetrueperpetrator(TP),andthereforeaprobability(D/N)(1p)D1thathewouldgiverisetoauniquematch,takingaccountoftheprobability(1p)D1thatnoadditionalmatcharisesbychanceforanyofthe(D1)innocentpersonsinthedatabase.Butthereisalsoaprobability(1D/N)pD(1p)D1thattheTPisnotinthedatabaseandthatexactlyoneoftheinnocentpersonsinthedatabasedoesmatch.Giventhefactthatauniquematchhasbeenfound,theprobabilitythatthisistheTPis1/(1+p(ND)).Thereisacomplementaryprobability(approximatelyp(ND)ifthisis1)thatthisisafalsematch,typicallymuchgreaterthantheprobabilityofafalsematchwhenasingletestiscarriedoutonaprimesuspect.3Thefactorbywhichthisriskisincreasedcomparedwiththeillustrativeexampleinthelastparagraphis(ND),thenumberofpotentialsuspectsoutsidethedatabasepotentially100,000ormoreinsomecases(Kaye2009).4OfcourseamatchfoundinaDNAtrawl(p.479) leadstoinvestigationofthesuspectandfurthernon-geneticevidence.Insomecasesthismayleadtoconfidentconviction(e.g.ifthesuspectturnsouttomatchaCCTVimagefromthecrimescene)oreliminationoracquittalifthematchisclearlyfalse(e.g.ifthereisadefinitealibi).ButDNAevidenceoftenappearstothejurytobethestrongestevidenceinacaseandconvictionfollowingaDNAtrawlmayevenresultwhenallotherevidenceseemstoweighinthedefendantsfavour(Donnelly2005;Kaye2009).ItiscrucialinsuchcasesthatajuryshouldrecognisethattheDNAdatacomesfromatrawlandcarriesamuchgreaterriskoffalseincriminationthanifitwereconfirmingsuspicionofaprimesuspect.Withoutthis,suchtrialsmustbeconsideredunsafe.

HowarewetoreconcilethisconclusionwiththeseeminglycontraryBayesianargumentpresentedbyDawid(2002)andKaye(2009)thattheorderinwhichDNAandnon-geneticdataareobtainedisimmaterial.Theyarguethatsinceatrawleliminatessuspectsaswellasidentifyingonewhomatches,whencombinedwithotherevidenceitwillrationallyleadtogreaterprobabilityofguiltthanwouldbeinferredifthesamepersonhadbeentestedastheprimesuspect.ThisargumentisillustratedinFigure17.1forthesimplecaseconsideredabove,wherethereisstrongnon-geneticevidence(E)againstadefendant,eitheremergingattheoutsettojustifyconfirmatoryDNAtestingofthedefendantasaprimesuspectorelseemergingafterthesuspecthasbeenidentifiedinatrawl.Thisshowshowtheoddsonthesuspectsguiltareaccumulatedtosimilaroutcomes,simplyinadifferentorder.Thisnear(p.480) equivalenceofoutcomeshasbeenusedtochallengethenotionthatjuriesneedtobeinformedofthespecialfactsand

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 13 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

statisticalconsiderationswhensuspectidentificationoccursthroughtrawlsormultipletesting(Balding2002,2005;Kaye2009).

Figure17.1 .ComparisonofBayesiantreatmentsofprobabilityofguilt.Arrowsshowsuccessivechangesinprobabilityofguiltandassociatedodds,forasuspectidentified(1)bynon-geneticevidence(E)followedbyaconfirmatoryDNAtest,or(2)throughtrawlinaDNAdatabasefollowedbyinvestigationanddiscoveryofE.Thenon-geneticevidenceonitsownleadstoaprobabilityofguilt(s)thatistakenheretobe50percent.N,Dandpareexplainedinthetext.Dashedarrowsshowthe(approximate)probabilityofaDNAmatchbeingfalsewitheachprocedure.Ifsuchamatchwereknowntobefalsethiswouldlead,givenassumptionsinthetext,tocertainacquittal.

Iwouldwhollyacceptthisargumentifthecollectionandevaluationofnon-geneticevidencewererigorous,quantitative,freeofselectionbiasanduncertainties,andderivedfrominvestigationofallpossibleavenues.Butthisisunrealistic,sometimesevenforevidencebackedbyscience.Therealityisthatmuchevidencedependsonhunchesandqualitativeargument,andthoroughinvestigationhasoftenbeenrestrictedtojustoneorafewofthepossiblesuspects.Thedegreeofsuspicionappropriateforaprimesuspectisoftenhighlyuncertain,andthediscoveryofapositiveDNAmatchprovidesstrongvindicationofthepotentiallyuncertainargumentsthatledtothestatusasprimesuspect.WhenasuspectisinsteadidentifiedbyDNAtrawl,theargumentsconstructedaroundnon-geneticevidenceobtainedafteridentificationwilllackthattestofauthenticityandmustretainalltheircaveatsanduncertainties.Contrastforexample,twosituations.5In(A),SbecomesasuspectbecauseanunreliableinformantassertsSdidit,SisDNAtestedandprovestomatchthecrimescene.In(B),SissuspectedbecausetrawlofadatabaserevealsthatheisaDNAmatchandthentheinformantaddshisassertionSdidit.Inneithercasewilltheinformantsblandassertioncarrysignificantweight,butitisessentialthatajuryshouldknowwhichscenarioappliestothecase,becausetheprobabilitythattheprocedurewouldproduceaDNAmatchincriminatinganinnocentpersonismuchgreaterin(B)than(A),byafactorequaltothenumberofplausiblesuspectswithinthedatabase.Aworthwhileapproachisforajurytoconsiderintrawlcaseswhether,ifsubsequentnon-geneticevidencehadbeenestablishedattheoutset,itwouldhavejustifiedthedefendantbeingtestedastheprimesuspect.OnlythenistheprobabilityoffalseincriminationbyDNAaslowastherandommatchprobabilityp.Incase(B)abovethisconsiderationwouldhighlightthecrucialimportanceofknowingwhethertheinformantspost-trawlassertionwasmadewithorwithoutknowledgethatSwasbythenasuspect.

Mythesishereisthatcourtsneedtoaddresstheprobabilitythatpoliceprocedures

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 14 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

(includingtrawlsforsuspects)couldhaveproducedevidenceagainstaninnocentpersonthatisatleastasincriminatingasthatpresented.Thisisaformofdataprobabilityandishowthecourtshouldultimatelyjudgethesafetyofaconviction.Itisajudgmentthattakesaccountofmorethanjustfactslinkingthedefendanttothecase,butsoitshould.Thedefendant(p.481) isincourtduetooperationoftheseprocedures,whichmustthereforebeunderscrutinyjustasmuchasthedefendanthimself.Ajudgmentthatfocusesonlyontheprobabilitythatthisdefendantisguiltywillnecessarilybedefi-cient.Itwouldessentiallyapportionprobabilitybetweenallpossiblesuspects,onlyoneofwhomisfullyinvestigatedandpresented,whilethetrueculpritandnatureofthecrimemaynotevenhaveenteredconsideration.Thereisastronganalogyheretothedifficultyofjudginghowlikelyitisthataparticularscientifichypothesisiscorrect:thisisalwaysuncertainbecausewesimplyneverknowifwehavehadtheluck,insightorimaginationtoconsiderthecorrectexplanationofthedataweobserve.

ConclusionAcknowledgementandcharacterisationofuncertaintiesarekeyelementsintheapplicationofknowledgeandevidence.Alackofawarenessofthenatureorextentofuncertaintycanleadtoconfusionandinappropriatedecisionsinareasofeducation,scienceandlaw.Myfocusherehasbeenonadistinctionthatcutsacrossthemajordebatesthatoftenemergeinstatistics(aboutthedefinitionofprobabilityinfrequentistorBayesianterms)andinlaw(aboutthemeritsofquantitativeorqualitativeapproachestodecisionmakingincourt).ThedistinctionIamconcernedwithisbetweenprobabilitiesassignedtohypothesesandprobabilitiesfortheobservationofdata,conditionalonaspecifichypothesis.Thereisnotachoicebetweenaddressingjustoneortheother:eachhasitsownlogicanditsdomainsofrigorousapplicationinbothscienceandlaw.Botharecentraltotheprocessofestablishingknowledge,whichitselfconsistsofgradationsofbeliefinhypothesesjustified,atleastpartially,byevidencefromdata.Thelinkbetweenhypothesisanddataprobabilitiesisseldomrigorousineitherscienceorlaw.Therearegreyareaswhereconclusionsmaybemattersofopinion.Scienceingeneralgetsroundthisproblembyrelyingontheindefiniteaccumulationofevidencethatwillultimatelyswampallbutthemostextremedifferencesofopinion.Criminallawreliesmoreonthemoderatingeffectofajurydecisiontogenerateanoutcomethatisgenerallyacceptedasreasonable.Inthelegalcontexthowever,Iarguethattheusualfocusonprobabilitythatthedefendantisguilty(ahypothesisprobability)isinadequate.Clearbenefitsariseifaverdictisconsideredtobeultimatelyconstrainedbyadataprobability:howlikelyisitthatsuchincriminatingevidencecouldhavearisenforaninnocentperson?

(p.482) Note.IamgratefultoD.KayeandD.Baldingandtotheeditorsofthisvolumeforconstructivecomments.

References

Bibliographyreferences:

Anderson,T.,Schum,D.andTwining,W.(2005),AnalysisofEvidence,2ndedn.(Cambridge,CambridgeUniversityPress).

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 15 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

Balding,D.J.(2002),TheDNAdatabasesearchcontroversy,Biometrics,58:2414.

Balding,D.J.(2005),Weight-of-EvidenceforForensicDNAProfiles(Chichester,JohnWiley).

Cohen,L.J.(1977),TheProbableandtheProvable(Oxford,ClarendonPress).

Dawid,A.P.(1986),ProbabilityForecasting,inEncyclopediaofStatisticalSciences,vol.7,ed.S.Kotz,N.L.JohnsonandC.B.Read(NewYork,Wiley-Interscience),pp.21018.

Dawid,A.P.(2002,)BayessTheoremandweighingevidencebyjuries,inBayessTheorem,ed.R.Swinburne,Proc.Brit.Acad.113:7190

Devlin,B.(2000),TheevidentiaryvalueofaDNAdatabasesearch,Biometrics,56:1276

Donnelly,P.(2005),Appealingstatistics,Significance,2:468.

Fisher,R.A.(1925),StatisticalMethodsforResearchWorkers(Edinburgh,OliverandBoyd).

Gardner-Medwin,A.R.(1995),Confidenceassessmentintheteachingofbasicscience,AssociationforLearningTechnologyJournal,3:805.

Gardner-Medwin,A.R.(2005),Whatprobabilityshouldajuryaddress?,Significance,2:912.Availableonlineat.

Gardner-Medwin,A.R.(2006),Confidence-basedmarkingtowardsdeeperlearningandbetterexams,inInnovativeAssessmentinHigherEducation,ed.C.BryanandK.Clegg(London,Routledge,TaylorandFrancisGroup).

Kaye,D.(2009),Roundinguptheusualsuspects:alegalandlogicalanalysisofDNAdatabasetrawls,NorthCarolinaLawReview,87:425503.

Kempthorne,O.andFolks,L.(1971),Probability,StatisticsandDataAnalysis(Ames,IA,IowaStateUniversityPress).

Lindley,D.V.(1972),BayesianStatistics:areview(Philadelphia,PA,SocietyforIndustrialandAppliedMathematics).

MacKay,D.J.C.(2003),InformationTheory,InferenceandLearningAlgorithms(Cambridge,CambridgeUniversityPress).

NationalResearchCouncilCommitteeonDNAForensicScience(1996),AnUpdate:theevaluationofDNAforensicDNAevidence(Washington,DC,NationalAcademyPress).

Popper,K.(1959),TheLogicofScientificDiscovery(London,Hutchinson).

Roberts,P.andZuckerman,A.(2004),CriminalEvidence(Oxford,OxfordUniversity

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 16 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

Press).

Royall,R.M.(1997),StatisticalEvidence:alikelihoodparadigm(London,ChapmanandHall).

RoyalStatisticalSociety(2002),LetterfromthePresidenttotheLordChancellorregardingtheuseofstatisticalevidenceincourtcases..

Senn,S.(2003),DicingwithDeath:chance,riskandhealth(Cambridge,CambridgeUniversityPress).

Song,Y.S.,Patil,A.,Murphy,E.E.andMontgomery,S.(2009),AverageprobabilitythataColdHitinaDNAdatabasesearchresultsinanerroneousattribution,J.Forensic.Sci.,54:227.

Stockmarr,A.(1999),LikelihoodratiosforevaluatingDNAvidencewhenthesuspectisfoundthroughadatabasesearch,Biometrics,55:6717.

Tillers,P.andGottfried,J.(2006),Acollateralattackonthelegalmaximthatproofbeyondareasonabledoubtisunquantifiable,Law,ProbabilityandRisk,5:13557.

vonNeumann,J.andMorgenstern,O.(1944),TheoryofGamesandEconomicBehavior(Princeton,NJ,PrincetonUniversityPress).

Notes:ProceedingsoftheBritishAcademy,171,465483.TheBritishAcademy2011.

(1)Forexample,measurementsthatarenormallydistributedrandomvariableswithmean,wherethepriorprobabilitydistributionforisessentiallyuniformoverarangemuchlargerthanthemeasurementstandarddeviation.

(2)Ifthepriorprobabilityofguiltofthematchingsuspectiss,basedonnon-geneticevidenceandeliminationofanypreviouslytestedsuspects,thentheprobabilitythatthismatchisfalseisapproximatelyp(1-s)/s(assumingthisitselfissmall).Evenifsfortheprimesuspectisconsideredunquantifiable,butthereareatmostMsuspectswhoareatallplausible,smustrationallybetakenasatleast1/Monthebasisthathisprobabilityofguiltmustbeconsideredatleastashighaseachoftheothers.Thissetsanupperboundontheprobabilityofafalsematchasp(M1).

(3)AdifferentderivationofthesameconclusionisgivenbySongetal.(2009).

(4)Notethatalargerandmorerelevantdatabase(largerD)reducestheriskthatauniquematchwillbeafalsematch.ThiscanbeseenasanargumentinfavourofmaintaininglargeDNAdatabases.Largedatabaseswouldalsoincreasethefrequencywithwhichdatabaseusewouldgeneratesuspectsforinvestigation.However,theremightbelegitimateconcernsthatinappropriatehandlingincourtofthestatisticalandtechnicalissuessurroundingdatabaseusemightleadtomorefalseconvictions.

Reasonable Doubt: Uncertainty in Education, Science and Law

Page 17 of 17

PRINTED FROM BRITISH ACADEMY SCHOLARSHIP ONLINE (britishacademy.universitypressscholarship.com). (c) CopyrightBritish Academy, 2014. All Rights Reserved. Under the terms of the l icence agreement, an individual user may print out a PDF of asingle chapter of a monograph in BASO for personal use (for detailssee: http://britishacademy.universitypressscholarship.com/page/privacy-policy/privacy-policy-and-legal-notice ). Subscriber: PontificiaUniversidad Catolica del Peru (PUCP); date: 30 April 2015

(5)IamgratefultoD.Kayeforsuggestingconsiderationofthisscenario.

Accessbroughttoyouby: PontificiaUniversidadCatolicadelPeru(PUCP)