Risk-Limiting Audits: Lessons Learned - MIT Election...
Transcript of Risk-Limiting Audits: Lessons Learned - MIT Election...
![Page 1: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/1.jpg)
Risk-LimitingAudits:LessonsLearnedNealMcBurnett
Atthe2018-12-07MITElectionAuditSummit
![Page 2: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/2.jpg)
NealMcBurnettConsultantColoradoRLAprojectteammemberwithDemocracyWorksandFree&FairSpeakingformyself
Workingonelectionauditsandintegritysince2003
PollworkerVolunteerIEEEP1622ViceChairCenterforElectionScienceBoardmemberSoftwaredeveloperConsultant
History,Links:
TheColoradoRisk-LimitingAuditProject(CORLA)Pilotaudits:earlyandoften
![Page 3: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/3.jpg)
BestPractice:Ballot-LevelRLAsWithagoodsystems,processesanddata,youcandoballot-levelrisk-limitingauditswhichlimittheriskthattabulationerrorsorattacksresultingettingthewrongoutcome
Inhundredsofcontests,dozensofcountiesAcrossoverlappingdistrictsinastateEfficiently:auditedlessthantenthousandballotsstatewide
Simultaneously,opportunistically,gatherevidenceonandreportrisklevelsforalltherestofthecontests(2018stillworkinprogress)
Colorado'snewstatewidesystemisamongthemostcosteffectiveandbestforauditing:central-countscannerswithBMDsavailableforaccessibility
![Page 4: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/4.jpg)
ResourcesAvailableFourvendorspresentedandpilotedsystemsthatcoulddoballot-levelcomparisonRLAsin2015:centralcountscanningsystemsfromDominion,Hart,ClearBallot,ES&S
OpensourceColoradoRLAcodeavailabletosupporttheseaudits,continuestobeenhanced
![Page 5: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/5.jpg)
SupportforRLAsWidespread,transpartisanconsensusonneedforbothpaperballotsandaudits.
2003Four-partyconsensusinBoulderColorado2017EAC/NISTVoluntaryVotingSystemGuidelines(VVSG)2.02018SecuringtheVote:ProtectingAmericanDemocracy|TheNationalAcademiesPress
Hugestepsforward,stillmuchtodo
Whyisittakingsolongtoadoptrobustaudits?
ElectionsareincreasinglycomplicatedYoucan'teasilyauditthedatayou'vegotYoucan'teasilygetthedatayouneedCriticalCommonDataStandardsworkbyEAC/NIST
![Page 6: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/6.jpg)
CommonDataFormatsWeneedformatstandards!EAC/NIST
JohnWack:OverviewofVVSG-InteroperabilityCommonDataFormats(twopresentations)
ElectionResultsCDFV1publishedasSP1500-100.
UsedinOH,NC,LACounty,otherstatesinprogress.V2synchronizeswithGoogle/VIP5.1,addsJSON.ElectionLogExportCDFsoonpublishedasSP1500-101.VoterRecordsInterchangeCDFslatedforreviewbyVRvendorsandthenpublishedasSP1500-102.
InitialuseinOHandbyOSET.CastVoteRecordsCDFschematobepublishedasSP1500-103.Continueddevelopmentanddocumentationofelectionprocessbusinessmodelsandvotingmethoddescriptions.
![Page 7: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/7.jpg)
EvidencepresentedandcheckedPublicRLAOversightProtocol,StephanieSinger,NealMcBurnett2017
Elements:
1ChainofCustody2Tabulation3Manifest4Commitment5Randomselection6Ballotcardretrieval7BallotInterpretationanddataentry8Endingtherandomselectionandexaminationofballotscards9HandCount10AuditConclusionsAffectOutcomes
![Page 8: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/8.jpg)
ConvincingOfficialsofElectionOutcomesColoradoRLAincludesrla_exporttooltoprovidenecessarydataforOversightProtocolincsv/jsonformats
rla_reportdemonstrationcodeinprogresstopreciselyexplainandimplementoversightsteps.
Verifiersshouldofcourseimplementorvettheirownprocesses,code,etc.
Level1:ElectionAdminstrators
Coloradocountiesandstate,basedontheirknowledgeoftheCVRsetc,dramaticallylimitedtheriskofanincorrecttabulationoutcomeinhundredsofcontests
Farmorethanmoststatescansay,veryefficient!
![Page 9: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/9.jpg)
ConvincingOthersofElectionOutcomesLevels2and3:LosersandthePublic
MuchmoretransparencythaninthepastStillseveralcrucialholesleftinoversightprotocolSomesummarydatanotavailableyet
Wrestlingwithballotanonymityissues=>noCVRs
Can'tchecktotals,interpretations,etc.
Nevertheless,appreciateamazingongoingaccomplishmentsbystateandcountiesunderverychallengingcircumstances!
Moretocome!
![Page 10: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/10.jpg)
DiscrepancyInvestigationsDetailedreportsstillin-progress
SoftwareshouldinformAuditBoardofeachdiscrepancyrightafterentry.Thatwouldhelpinvestigation,qualitycontrolfeedback,andtrustintheprocess.
PoorviVoraandIhaveadocumentinprogressonhowtoinvestigatediscrepancies,preservingintegrity,efficiency,flexibility
Moredatawouldbeinvaluableinfinetuningtheprocess
![Page 11: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/11.jpg)
RemainingChallenges1Enhancereportingconvenienceandanalysis
Discrepancies,risklevelsforopportunisticallyauditedcontestsinarbitrarymixofcountiesAutomaticgeneration,publicationofAuditCenterforpublic
Modularizesoftwareforusewithexternalstatisticalmodules,additionalmethods(SUITE,Bayesian,etc.)
HandleIn-Precinct/Votecenterscanners,whichrandomizeballotsand/orCVRs:complicateprocessofmatchingpaperballotwithCVR
Supportbatch-comparisonaudits(generallypredictableriskreduction),ballot-pollingaudits(unpredictable,impracticalfortightmargins)
![Page 12: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/12.jpg)
RemainingChallenges2Fostercollaborationbetweenclerks,privacyexperts,toolsmithsaroundpreservinganonymity,especiallyforcomplicatedsituationinColorado
Auditmoresystems:VR,signatureverification,envelopesorters
![Page 13: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/13.jpg)
TargetedauditsEncouragecandidates,publictoidentifyadditionalinterestingballotstotargetforauditing
InadditiontofullrandomselectionauditCouldbechosenbasedonCVRs,markdensitydata,ballotimages
![Page 14: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/14.jpg)
PublicengagementinverificationPromotepublicparticipationinauditPrintballottrackingpageswithQRcodesApptophotographballot+QRcodeAssistpublictweetslike"Iverifiedthisvote"
![Page 15: Risk-Limiting Audits: Lessons Learned - MIT Election Labelectionlab.mit.edu/sites/default/files/2018-12/eas-mc... · 2019. 12. 12. · Election Log Export CDF soon published as SP](https://reader035.fdocuments.in/reader035/viewer/2022071420/6119a446d76d8963921e1327/html5/thumbnails/15.jpg)
AcknowlegementsPaulTigerJoePezzilloPaulWalmsleyRonRivestPhilipStarkHarvieBranscombHillaryHallandBoulderCountyteamElectionVerificationNetworkColoradoLegislatorsIncrediblededicationofSOSstaffandClerks!Free&FairTeambeyondthecallofdutyGaloissupport
Updatedslides:http://bcn.boulder.co.us/~neal/elections/audit-summit.pdf