Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common...
-
Upload
hugh-thomas-preston -
Category
Documents
-
view
217 -
download
0
Transcript of Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common...
![Page 1: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/1.jpg)
Batch Reports for Audits - ElectionAudits and the Boulder
2008 Election
Neal McBurnett NIST Common Data Formats
Workshop Oct 29 2009
![Page 2: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/2.jpg)
Questions
Ask questions any time Who has been involved with an audit? Who knows of a jurisdiction where ballots are
reconciled by ballot style as well as by precinct?
![Page 3: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/3.jpg)
Effective audits are critical
Improve election processes Attain software independence and
confidence in elections
![Page 4: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/4.jpg)
Humboldt County 2008Paper ballots, optical scan
197 ballots deleted without a trace
![Page 5: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/5.jpg)
Discovered later by Humboldt County Election Transparency Project audit
Kudos to Mitch Trachtenberg, Registrar of Elections Carolyn Cernich, and colleagues
![Page 6: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/6.jpg)
Goal:
software independence(Rivest & Wack)
via auditable paper records,good audits
![Page 7: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/7.jpg)
Audit:
Compare system's reported results with the evidence
Take sample of detailed results by audit unitand compare to hand counts of those audit
units
![Page 8: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/8.jpg)
Auditing steps`
Produce report by audit unit (precinct or batch)
Reconcile number of ballots in each unit Recalculate sum of subtotals Randomly select audit units to audit Count audit units by hand Compare results Continue auditing until statistical evidence
meets goal (limit risk of wrong outcome)
![Page 9: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/9.jpg)
Report of Vote Counts by Audit Unit
Audit Unit Susan Nelson Under Over Total100 137 144 2 0 383101 77 68 1 0 247102 122 87 3 2 316103 98 102 1 1 305104 22 18 2 0 146105 103 140 1 0 349
Total 559 559 10 3 1746
![Page 10: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/10.jpg)
Must include undervotes and overvotes in the results for each audit unit. For multiple-
winner contests, undervotes are ambiguous, so also need a count of ballots
on which the contest appears
![Page 11: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/11.jpg)
Principles and Best Practices forPost-Election Audits (2008)
http://electionaudits.org/principles
League of Women VotersElection Audit Report (2009)
![Page 12: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/12.jpg)
Check out typical Hart precinct report
PDFNo under/over
By precinctBig numbers
Not suitable for auditing in many places - vulnerable to many attacks!
Inefficient everywhere
![Page 13: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/13.jpg)
From NIST's Call for Papers:
“Human-readable versus machine readable-human-readability matters for transparency
and auditing”
Not true – auditing requires aggregation and summing subtotals to confirm totals.
The most effiecient audits involve analyzing vote-count data for all audit units
Standard format helps with targeting units
![Page 14: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/14.jpg)
Current reports typically only by precinct
Auditing by precinct is outmoded, ineffecient and problematic
![Page 15: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/15.jpg)
Agreement on need for standardizedsmall-batch audit reporting from meeting
hosted by American Statistical Association, October 2009
![Page 16: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/16.jpg)
Batch reporting important:
Organization of the paper
Privacy of provisional ballots
Small batches more efficient to audit
Individual ballot interpretations needed for emerging methods that audit at the ballot
level without breaching confidentiality
![Page 17: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/17.jpg)
Organization of the paper:
Reports are by precinct
But often piles of paper aren't
- Mail in- Early voting- Provisional
![Page 18: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/18.jpg)
Reporting by precinct before and after provisional ballots are included can
compromise the privacy of the provisional voters
Instead, report provisionals in batches, and precincts only after all provisionals are in
![Page 19: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/19.jpg)
Significance of the results driven by how many batches you audit, not how many
ballots you count
Looking for incorrect vote counts
Don't care about total vote count for the sample
![Page 20: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/20.jpg)
Typically, amount of work required to perform a vote-tabulation audit is roughly proportional to the batch sizes, with no
reduction in statistical efficacy.
![Page 21: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/21.jpg)
Getting batch data much more frustrating than it should be
![Page 22: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/22.jpg)
Boulder County, 2008
170,000 ballots, 65 races13 contests audited, 40000 choices
Thanks to Clerk Hillary Hall and staff!
![Page 23: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/23.jpg)
Hart InterCivic system
No batch reports supported
Only 17% cast in precinct
67% mail-in, 16% early voting
![Page 24: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/24.jpg)
Solution: run cumulative report 525 times
Once for each batch
Subtract each report from the previous report to get batch sub-totals
![Page 25: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/25.jpg)
Like snapshots of election-night reporting through the night
But we want lots of them....
![Page 26: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/26.jpg)
Report of Cumulative Vote Countsby Audit Unit
Audit Unit Susan Nelson Under Over Total100 137 144 2 0 383101 214 212 3 0 630102 336 299 6 2 946103 434 401 7 3 1251104 456 419 9 3 1397105 559 559 10 3 1746
Total 559 559 10 3 1746
![Page 27: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/27.jpg)
Report of Vote Counts by Audit Unit
Audit Unit Susan Nelson Under Over Total100 137 144 2 0 383101 77 68 1 0 247102 122 87 3 2 316103 98 102 1 1 305104 22 18 2 0 146105 103 140 1 0 349
Total 559 559 10 3 1746
![Page 28: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/28.jpg)
No single Hart report had all the data we wanted in a reasonable format.
“Crystal Reports” XML schema reflected paper layout, not logical data
Didn't have ballot counts – dug those out of PDF reports
“CSV” reports didn't have columns that lined up
![Page 29: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/29.jpg)
Sequoia reports are clean and complete
Good “text with headings” report (tab-separated)
Samples at
http://bcn.boulder.co.us/~neal/electionaudits/formats/
![Page 30: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/30.jpg)
Reports also frequently take a long time to produce – painful when doing hundreds of
times.
![Page 31: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/31.jpg)
Audit results also need to be published in a standard format, suitable for aggregation
across jurisdictions
Include hand counts and discrepancies for all units audited, whether they were
targeted or randomly selected, information needed to validate the selections (e.g.
algorithms used, random seeds used as input) , results of the audit
![Page 32: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/32.jpg)
Audit-related report objects
Auditable vote count report Audit report Random seed generation Set of audit unit results Audit unit result
![Page 33: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/33.jpg)
Audit unit result
Audit unit id Hand count time, personnel assigned For each candidate or “under” or “over”
System tally Hand count (multiple?) Discrepancies (Hand count - system) Resolution code Resolution text description
![Page 34: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/34.jpg)
Audit report objects
audit report random seed generation could be for a series of selections or an individual roll of dice for one audit unit
![Page 35: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/35.jpg)
ElectionAudits software is open source, BSD license
Supports the Best Practices
Automates many steps of the audit
Enter the data, publish with statisticsRoll 15 dice, publish all the selections
To do: automate discrepancies, escalation
![Page 36: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/36.jpg)
Look at some real data from Boulder 2008
![Page 37: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/37.jpg)
Features of ElectionAudits Imports standard election report files: Hart,
Sequoia Produces auditable reports for the public. Protects voter anonymity by merging small audit
units Doesn't require that paper ballots be sorted into
piles by precinct Can produce batch reports from sequence of
cumulative reports Facilitates risk-limiting audits Verifiably pseudo-random: dice + "Sum of Square
Roots"
![Page 38: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/38.jpg)
Biggest challenge: getting small-batch data out of election systems
![Page 39: Batch Reports for Audits - ElectionAudits and the Boulder 2008 Election Neal McBurnett NIST Common Data Formats Workshop Oct 29 2009.](https://reader035.fdocuments.in/reader035/viewer/2022062407/56649db65503460f94aa8ad4/html5/thumbnails/39.jpg)
Summary
Proper audits are critical Risk-limiting audits in Colorado law Small-batch reports help enormously Standard, machine-readable format is
needed EML looks appropriate http://launchpad.net/electionaudits