UnclearBallot:Automated Ballot Image Manipulation

Matthew Bernhard, Kartikeya Kandula, Jeremy Wink, and J. Alex Halderman

Department of Electrical Engineering and Computer Science, University of Michigan

{matber, kartkand, jreremy, jhalderm}@umich.edu

Abstract. As paper ballots and post-election audits gain increasedadoption in the United States, election technology vendors are offeringproducts that allow jurisdictions to review ballot images—digital scansproduced by optical-scan voting machines—in their post-election auditprocedures. Jurisdictions including the state of Maryland rely on suchimage audits as an alternative to inspecting the physical paper ballots.We show that image audits can be reliably defeated by an attacker whocan run malicious code on the voting machines or election managementsystem. Using computer vision techniques, we develop an algorithmthat automatically and seamlessly manipulates ballot images, movingvoters’ marks so that they appear to be votes for the attacker’s preferredcandidate. Our implementation is compatible with many widely usedballot styles, and we show that it is effective using a large corpus ofballot images from a real election. We also show that the attack can bedelivered in the form of a malicious Windows scanner driver, which wetest with a scanner that has been certified for use in vote tabulationby the U.S. Election Assistance Commission. These results demonstratethat post-election audits must inspect physical ballots, not merely ballotimages, if they are to strongly defend against computer-based attacks onwidely used voting systems.

Keywords: optical scan, paper ballots, image manipulation, drivers, imageprocessing

1 Introduction

Elections that cannot provide sufficient evidence of their results may fail toadequately gain public confidence in their outcomes. Numerous solutions havebeen posited to this problem [9], but none has been as elegant, efficient, andimmediately practical as post-election audits [21,25,39]. These audits—in par-ticular, ones that seek to limit the risk of confirming an outcome that resultedfrom undue manipulation—are one of the most important layers of defense forelection security [32].

Risk-limiting audits (RLAs) rely on sampling robust, independent evidencetrails created by voter-verified paper ballots. However, other types of post-election

Fig. 1. Attack overview — A voter’s paper ballot is scanned by a ballot tabulator,producing a digital image. Malware in the tabulator—in our proof-of-concept, a micro-driver that wraps the scanner device driver—alters the ballot image before it is countedor stored. A digital audit shows only the manipulated image.

audits are gaining popularity in the marketplace. In particular, Clear Ballot,an election technology vendor in the United States, pioneered audit softwaredesigned to perform audits of images of ballots which have been scanned andtabulated, which we shall refer to as “image audits”. Other vendors have adoptedsupport for this kind of audit, and one U.S. state, Maryland, relies on imageaudits to provide assurances of its election results [33].

While image audits can help detect human error and aid in adjudicatingmismarked ballots, we show that they cannot provide the same level of securityassurance as audits of physical ballots. Since ballot images are disconnected fromthe actual source of truth—physical paper ballots—they do not necessarily providereliable evidence of the outcome of an election under adversarial conditions.

In this paper, we present UnclearBallot, an attack that defeats image auditsby automatically manipulating ballot images as they are scanned. Our attackleverages the same computer vision approaches used by ballot scanners to detectvoter selections, but adds the ability to move marks from one target area toanother. Our method is robust to inconsistent or invalid marks, and can beadapted to many ballot styles.

We validate our attack against a corpus of over 180,000 ballot images fromthe 2018 election in Clackamas County, Oregon, and find that UnclearBallot canmove marks on 34% of the ballots while leaving no visible anomalies. We alsotest our attack’s flexibility using six widely used styles of paper ballots, and itsrobustness to invalid votes using an established taxonomy of voter marks. As aproof-of-concept, we implement the attack in the form of a malicious Windowsscanner driver, which we test using a commercial-off-the-shelf scanner certifiedfor use in elections by the U.S. Election Assistance Commission.

UnclearBallot illustrates that post-election audits in traditional voting systemsmust involve rigorous examination of physical ballots, rather than ballot images,if they are to provide a strong security guarantee. Without an examinationof the physical evidence, it will be difficult if not impossible to assure thatcomputer-based tampering has not occurred.

The remainder of this paper is organized as follows: Section 2 providesbackground on image audits, ballot scanners, and image processing techniqueswe use to implement our attack. Section 3 describes the attack scenarios against

2

Fig. 2. Terms for parts of a marked ballot, following Jones [23].

optical scanners and image audits. Section 4 explains the methodology of ourattack. In Section 5 we present data indicating that our attack can be robust tovarious ballot styles and voter marks. Section 6 contextualizes our attacks anddiscusses mitigations. We conclude in Section 7.

2 Background

Our attack takes advantage of two aspects of optical scanner image audits: thescanning and image processing techniques used by scanners, and the reliance onscanned images by image audits. Here we provide a brief discussion of both.

2.1 Ballot Images

Jones [23] put forth an analysis of the way that ballot scanners work, particularlythe mark-sense variety that is most common today. All optical scanners currentlysold to jurisdictions, as well as the vast majority of scanners used in practice in theU.S., rely on mark-sense technology [44]. Scanners first create a high-resolutionimage of a ballot as it is fed past a scan head. Software then analyzes the imageto identify dark areas where marks have been made by the voter.1 Once markshave been detected, systems may use template matching to translate marks intovotes for specific candidates, typically relying on a barcode or other identifier onthe ballot that specifies a ballot style to match to the scanned image.

Detecting and interpreting voter marks can be a difficult process, as votersexhibit a wide range of marking and non-marking behavior, including not filling intargets all the way, resting their pens inside targets, or marking outside the target.The terms Jones developed to refer to the ballot and marks are illustrated inFigure 2. Marks that adequately fill the target and are unambiguously interpretedas votes by the scanner are called reliably sensed marks, and targets that areunambiguously not filled and therefore not counted are reliably ignored marks.

1 The details of how marks are identified vary by hardware and scanning algorithm.See [13] for an example.

3

Fig. 3. Taxonomy of voter marks adapted from Bajcsy [2], including the five leftmostmarks that may be considered marginal marks.

Marks of other types are deemed marginal, as a scanner may read or ignore them.Moreover, whether a mark should be counted as a vote is frequently governed bylocal election statute, so some marginal marks may be unambiguously countedor ignored under the law, even if not by the scanner.

Bajcsy et al. [2] further develops a systematization of marginal marks anddevelops some improvements on mark-detection algorithms to better accountfor them. An illustration of Bajcsy et al.’s taxonomy is shown in Figure 3. Jiet al. [22] discuss different types of voter marks as applied to write-in votes, aswell as developing an automated process for detecting and tabulating write-inselections.

2.2 Image Audits

Risk-limiting post-election audits rely on physical examination of a statisticalsample of voter-marked ballots [24, 26, 39, 40]. However, this can create logisticalchallenges for election officials, which has prompted some to propose relaxations totraditional audit requirements. To reduce workload, canvass audits and recountsin many states rely on retabulation of ballots through optical scanners (see the2016 Wisconsin recount, for example [31]).

Some election vendors take retabulation audits a step further: rather thanphysically rescan the ballots, the voting system makes available images of all theballots for independent evaluation after the election [15, 16, 42].2 While the exactproperties of these kinds of image audits vary by vendor, they typically rely onautomatically retabulating all or some images of cast ballots, as well as electronicadjudication for ballots with marginal marks. These “audits” never examine thephysical paper trail of ballots, which our attack exploits.

Several jurisdictions have relied on these image audits, including Cambridge,Ontario, which used Dominion’s AuditMark [17], and the U.S. state of Maryland,which uses Clear Ballot’s ClearAudit [28]. Maryland has also codified imageaudits into its election code, requiring that an image audit be performed afterevery election [27].

2 While the review is made available to the public, the actual images themselves areseldom published in full out of concern for voter anonymity.

4

3 Attack Scenarios

Elections in which voters make their selections on a physical ballot are frequentlyheld as the gold standard for conducting a secure election [32]. However, theproperty that contributes most to their security, software independence [34],only exists if records computed by software are checked against records thatcannot be altered by software without detection. Image audits enable electionofficials to view images of ballots and compare them with the election systems’representation of the particular ballot they are viewing (called a cast vote recordor CVR). While these two trails of evidence may be independent from each other(for example, Clear Ballot’s ClearAudit [15] technology can be used to audit atabulation performed by a different election system altogether), they are notsoftware independent. A clever attacker can exploit the reliance on software byboth evidence trails to defeat detection.

To surreptitiously change the outcome of the election in the presence of animage audit, the attacker must alter both the tabulation result as well as the ballotimages themselves. Researchers have documented numerous vulnerabilities thatwould allow an attacker to infect voting equipment and change tabulation results(see [10, 20, 30] among others), so we focus on the feasibility of manipulatingballot images once an attacker has successfully infected a machine where theyare stored or processed.

The most straightforward attack scenario occurs when the ballot images arecreated by the same equipment that produces the CVR. In this case, the attackercan simply infect the scanner or tabulator with malware that corrupts both theCVR and the images at the same time. The attack could change the image beforethe tabulator processes it to generate the CVR, or directly alter both sets ofrecords.

In some jurisdictions, the ballot images that are audited are collected in aseparate process from tabulation—that is, by scanning the ballots again, as inMaryland’s use of ClearAudit from 2016 [28]. In this case, the adversary has toseparately attack both processes, and has to coordinate the cheating to avoidmismatches between the initial tally and the altered ballot images.

Depending on the timing of the audit, manipulation of ballot images neednot be done on the fly. For example, if the ballot images are created duringtabulation but the image audit does not occur until well after the election, anattacker could modify the ballot images while they are in storage.

For ease of explication, the discussion that follows assumes that ballot imagesare created at the time of tabulation, in a single scan. The attack we developtargets a tabulation machine and manipulates each ballot online as it is scanned.

4 Methodology

To automatically modify ballot images, an attacker can take a few approaches.One approach would be to completely replace the ballot images with ballotsfilled in by the attacker. However, this risks being detected if many ballots have

5

the same handwriting, and requires sneaking these relatively large data files intothe election system without being detected. For these reasons, we investigate analternative approach: automatically and selectively doctoring the ballot scans tochange the vote selections they depict.

For the attack to work successfully, we need to move voter marks to othertargets without creating visible artifacts or inconsistencies. We must be able todynamically detect target areas and marks, alter marks in a way that is consistentwith the voter’s other marks, and do so in a way that is undetectable to thehuman eye. However, there is a key insight that works in the adversary’s favor:an attacker seeking to alter election results does not have to be able to changeall ballots undetectably, only sufficiently many to swing the result. This meansthat the attacker’s manipulation strategy is not required to be able to changeevery mark—it merely has to reliably detect which marks it can safely alter andchange enough of them to decide the election result.

4.1 Reading the ballot

To interpret ballot information, we rely on the same techniques that ballotscanners use to convert paper ballots into digital representations. Attackers haveaccess to the ballot templates, as jurisdictions publish sample ballots well aheadof scheduled elections. Using template matching, an attacker does not have toperform any kind of sophisticated character recognition, they simply have to findtarget areas and then detect which of the targets are filled.

Our procedure to read a ballot is illustrated in Figure 4. First, we performtemplate matching to extract each individual race within a ballot. Next, weuse OpenCV’s [11] implementation of the Hough transform to detect straightlines that separate candidates and break the race into individual panes for eachcandidate. Notably, the first candidate in each race may have the race title andextra information in it (see Figure 4c), which is cropped out based on whitespace.

Target areas are typically printed on the ballot as either ovals or rectangles.To detect them, we construct a bounding box around the target by scanninghorizontally from the left of the race and then vertically from the bottom up,and compute pixel density values. The bounds are set to the coordinates wherethe density values first increase and last decrease. Once we have detected allthe target areas, we compute the average pixel density of the area within thebounding box to determine whether or not a target area is marked. We then useour template to convert marks into votes for candidates.

4.2 Changing marks

Once we have identified which candidate was marked by the voter, we can movethe mark to one of the other target locations we identified. If the vote is for acandidate the attacker would like to receive fewer votes—or if it is not a vote for acandidate they would like to win—the attacker can simply swap the pixels withinthe bounding boxes of the voter’s marked candidate and an unmarked candidate.

6

Fig. 4. Ballot manipulation algorithm — First, (a) we apply template matching toextract the race we intend to alter. Then, (b) we use Hough line transforms to separateeach candidate. If the first candidate has a race title box, (c) we remove it by computingthe pixel intensity differences across a straight line swept vertically from the bottom.For each candidate, (d) we identify the target and mark (if present) by doing four linearsweeps and taking pixel intensity. Finally, (e) we identify and move the mark. At eachstep we apply tests to detect and skip ballots where the algorithm might leave artifacts.

7

Original Manipulated

Fig. 5. Automatically moving voter marks — UnclearBallot seamlessly movesmarks to the attacker’s preferred candidate while preserving the voter’s marking style.It is effective for a wide variety of marks and ballot designs. In the examples above,original ballot scans are shown on the left and manipulated images on the right.

By moving marks on each ballot separately, we ensure that the voter’s particularstyle of filling in an oval is preserved and consistent across the ballot. Figure 5shows some marks swapped by our algorithm, and how the voters original markis completely preserved in the process.

4.3 UnclearBallot

To illustrate the attack, we created UnclearBallot, a proof-of-concept implemen-tation packaged as a malicious Windows scanner driver, which consists of 398lines of C++ and Python. We tested it with a Fujistu fi-7180 scanner (shown inFigure 6), which is federally certified for use in U.S. elections as part of ClearBallot’s ClearVote system [43]. These scanners are typically used to handle smallvolumes of absentee ballots, and must be attached to a Windows workstationthat runs the tabulation software.

The UnclearBallot driver wraps the stock scanner driver and alters imagesfrom the scanner before they reach the election management application. Wechose this approach for simplicity, as the Windows driver stack is relatively easy

8

Fig. 6. The Fujitsu fi-7180 scanner weused to test our attack has been certified bythe U.S. Election Assistance Commission foruse in voting systems. Our proof-of-conceptimplementation is a malicious scanner driverthat alters ballots on the fly.

to work with, but the attack could also be implemented at other layers of thecomputing stack. For instance, it could be even harder to detect if implementedas a malicious change to the scanner’s embedded firmware. Alternatively, it couldcould be engineered as a modification to the tabulation software itself.

Once a ballot is scanned, the resulting bitmap is sent to our image processingsoftware, which manipulates the ballot in the way described in Section 4.1. Priorto the election, the attacker specifies the ballot template, which race they wouldlike to affect, and by how much. While ballots are being scanned, the softwarekeeps a running tally of the actual ballot results, and changes ballot images onthe fly to achieve the desired election outcome. To avoid detection, attackers canspecify just enough manipulated images so that the race outcome is changed.

5 Evaluation

We evaluated the performance and effectiveness of UnclearBallot using two setsof experiments. In the first set of experiments, we marked different ballot stylesby hand using types of marks taxonomized by Bajcsy et al. [2]. In the second setof experiments, we processed 181,541 ballots from the 2018 election in ClackamasCounty, Oregon.

5.1 Testing Across Ballot Styles

In order for our application to succeed at its goal (surreptitiously changingenough scanned ballots to achieve a chosen election outcome), it must be ableto detect marks that constitute valid votes as well as distinguish marks whichwould be noticeable if moved. The marks in the latter case represent a larger setthan just marginal marks, as they may indeed be completely valid votes, butconsidered invalid by our mark-moving algorithm. For example, if we were toswap the targets on a ballot where the user put a check through their target, wemay leave a significant percentage of the check around the original target whenswapping. The same applies for marked ballots where the filled in area extendsinto the candidate’s name, which could lead our algorithm to swap over parts ofthe candidate’s name when manipulating the image.

9

Fig. 7. Ballots Styles — We tested ballot designs from five U.S. voting system vendors:Clear Ballot, Diebold, Dominion, ES&S, and Hart (two styles, eScan and Verity).

To detect anomalies for invalid ballots, we leverage the same intensity checkingalgorithm that first found the marked areas. The program checks if the width orheight is abnormally large, which would indicate an overfilled target, as well as ifthere are too few or too many areas of high intensity, which would indicate notarget or too many targets are filled out. If the program detects an invalid ballot,it will not be modified by the program.

To show our attack is replicable on a variety of different ballot styles, wemodified our program to work on six different sample ballot styles, shown inFigure 7. The ballots we tested come from the four largest election vendors inthe U.S. (ES&S, Hart InterCivic, Dominion, and Clear Ballot), as well as twoolder styles of ballots from Hart and Diebold.

Our first experiment was designed to characterize the technique’s effectivenessacross a range of ballot styles and with both regular and marginal marks. We

10

Ballot StyleInvalid Marks Valid Marks

Time/SuccessSkipped Success Failure Skipped Success Failure

Clear Ballot 55 5 0 26 34 0 25 msDiebold 60 0 0 6 54 0 11 msDominion 38 22 0 7 53 0 30 msES&S 52 8 0 29 31 0 54 msHart (eScan) 60 0 0 38 22 0 46 msHart (Verity) 60 0 0 27 33 0 21 ms

Table 1. Performance of UnclearBallot — We tested how accurately our softwarecould manipulate voter marks for a variety of ballot styles using equal numbers of invalidand valid marks. The table shows how often the system skipped a mark, successfullyaltered one, or erroneously created artifacts we deemed to be visible upon manualinspection. We also report the mean processing time for successfully manipulated races,excluding template matching.

prepared 720 marked contests, split evenly among the six ballot styles shownin Figure 7. For each style, we marked 60 contests with what Bajcsy [2] calls“Filled” marks, i.e. reliably detected marks that should be moved by our attack.We marked another 60 ballots in each ballot style with marginal marks, ten eachfor the five kinds of marginal marks shown in Figure 2 and ten empty marks.

Because the runtime of the template matching step of our algorithm is highlydependent on customization for the particular races on a ballot, we opted toskip it for this experiment. Rather than marking full ballots, we marked croppedraces from each ballot style and then ran them through our program. We thenmanually checked to ensure that the races the program moved were not detectableby inspection. Results for these experiments are shown in Table 1.

Despite rejecting some valid ballots, our program is still able to confidentlyswap a majority of valid votes. In a real attack, only a small percentage of voteswould need to actually be modified, a task easily accomplished by our program.Our program also correctly catches all votes that we have deemed invalid forswapping. This would make it unlikely to be detected in an image audit.

Dominion ballots saw a much higher rate of invalid mark moving, and Dieboldand Dominion ballots saw a much higher rate of valid mark moving. This islikely due to the placement of targets: on the Dominion ballots, the mark is rightjustified, separating it significantly from candidate label information, as can beseen in Figure 7. Similarly, the Diebold ballot provides more space around thetarget and less candidate information that can be intercepted by marks, whichwould cause Unclear Ballot to skip moving the mark.

In an online attack scenario (such as if a human is waiting to see the outputfrom the scanner), the attacker needs to be able to modify ballot scans quicklyenough not to be noticed. Factors which might affect how quickly our programcan process and manipulate ballots include ballot style, layout, and type of mark.During the accuracy experiment just described, we collected timing data for

11

Original Manipulated

Fig. 8. Attacking Real Ballots — Using 181,541 images of voted ballots from Clacka-mas County, Oregon, we attempted to change voters’ selections for the ballot measureshown above. UnclearBallot determined that it could safely alter 34% of the ballots. Forreference, Measure 102 passed by a margin of 5%, well within range of manipulation [14].We inspected 1,000 of them to verify that the manipulation left no obvious artifacts.

successfully manipulated ballot, and report the results in Table 1. The resultsshow that after the target race has been extracted, the algorithm completesextremely quickly for all tested ballot styles. We present additional timing dataat the end of the following section.

5.2 Testing with Real Voted Ballots

To assess the effectiveness of UnclearBallot in a real election, we used a corpusof scans of 181,541 real ballots from the November 6, 2018, General Electionin Clackamas County, Oregon, which were made available by Election IntegrityOregon [18]. Like all of Oregon, Clackamas County uses vote-by-mail as itsprimary voting method, and votes are centrally counted using optical scanners.All images were Hart Verity-style ballots, as shown in Figure 7.

We selected a ballot measure that appeared on all the ballots (Figure 8) andattempted to change each voter’s selection. UnclearBallot rejected 20,117 (11%)of the ballots because it could not locate the target contest. We examined asubset of the rejected ballots and found that they contained glitches introduced

12

during scanning (such as vertical lines running the length of the ballot), whichinterfered with the Hough transform.

To simulate a real attacker, we configured UnclearBallot with conservativeparameters, so that it would only modify marks when there was high confidencethat the alteration would not be noticeable. As a result, it would only manipulatemarks that were nearly perfectly filled in. In most cases, marks that were skippedextended well beyond the target, but the program also skipped undervotes,overvotes, or mislabeled scans. Under these parameters, the program altered thetarget contest in 62,400 (34%) of the ballot images.

Two authors independently inspected a random sample of 1,000 altered ballotsto check whether any contained artifacts that would be noticeable to an attentiveobserver. Such artifacts might include marks which were unnaturally cut off,visible discontinuities in pixel darkness (i.e. dark lines around moved marks),and so on. If these artifacts were seen during an audit, officials might recheck allof the physical ballots and reverse the effects of the attack. None of the alteredballots we inspected contained noticeable evidence of manipulation.

We also collected timing data while processing Clackamas County ballots.Running on a system with a 4-core Intel E3-1230 CPU running at 3.40 GHz with64 GB of RAM, UnclearBallot took an average of 279 ms to process each ballot.For reference, Hart’s fastest central scanner’s maximum scan rate is one ballotper 352 ms [37], well above the time needed to carry out our attack.

These results show that UnclearBallot can successfully and efficiently manip-ulate ballot images to change real voters’ marks. Moreover, the alterations likelywould be undetectable to human auditors who examined only the ballot images.

6 Discussion and Mitigations

UnclearBallot demonstrates the need for a software-independent evidence trailagainst which election results can be checked. It shows that audits based onsoftware which is independent from the rest of the election system is still notsoftware independent. To date, the only robust and secure election technologythat is widely used is optical-scan paper ballots with risk-limiting audits basedon a robust, well-maintained, physical audit trail. However, image audits are notuseless, and here we discuss uses for them as well as potential mitigations for ourattack.

Uses for image audits. So long as image audits are not the sole mechanism forverifying election results, they do provide substantial benefits to election officials.Using an image audit vastly simplifies some functions of election administration,like ballot adjudication in cases where marks cannot be interpreted by scannersor are otherwise ambiguous. Image audits can be used to efficiently identify anddocument election discrepancies, as has occurred in Maryland where nearly 2,000ballots were discovered missing from the audit trail in 2016 [28]. Image auditsalso identified a flaw in the ES&S DS850 high speed scanner, where it was causingsome ballots to stick together and feed two at a time [29].

13

Another way to utilize image audits is a transitive audit. Methods likeSOBA [8] seek to construct an audit trail using all available means of electionevidence, rooting the audit in some verification of physical record. By usingphysical records to verify other records, like CVRs or ballot images, confidence inelection outcomes can be transitively passed on to non-physical audit trails. Thedrawback with this kind of audit is that it usually requires the same level of workas an RLA, plus whatever work is needed to validate the other forms of evidence.However, since ballot image audits already require a low amount of effort, theymay augment RLAs and provide better transparency into the auditing process.

Image audits are an augmentation and a convenience for election adminis-tration, however, and should not be viewed as a security tool. Only physicalexamination of paper ballots, as in a risk-limiting audit, can provide a necessarylevel of mitigation to manipulated election results.

End-to-end (E2E) systems. Voting systems with rigorous integrity propertiesand tamper resistance such as Scantegrity [12] and Pret a Voter [35] provide adefense to UnclearBallot. In Scantegrity, when individuals mark their ballots, aconfirmation code is revealed that is tied to the selected candidate. This enablesa voter to verify that their ballot collected-as-cast and counted-as-collected, asthey can look up their ballot on a public bulletin board. Since each mark revealsa unique code, moving the mark would match the code with the wrong candidate,so voters would be unable to verify their ballots. If enough voters complain, thismight result in our attack being detected.

Pret a Voter randomizes the candidate order on each ballot, which creates aslightly higher barrier for our attack, as an additional template matching stepwould be needed to ascertain candidate order. More importantly, the candidatelist is physically separated from the voter’s marks upon casting the ballot, somalware which could not keep track of the correct candidate order could notsuccessfully move marks to a predetermined candidate. Since the candidate orderis deciphered via a key-sharing scheme, malicious software would have to infect asignificant portion of the election system and act in a highly coordinated way toreconstruct candidate ordering. Moreover, as with Scantegrity, votes are publishedto a public bulletin board, so any voter could discover if their vote had not beencorrectly recorded.

Other E2E systems which make use of optical scanning and a bulletin board,like STAR-Vote [6], Scratch and Vote [1], and VeriScan [7], are similarly protectedfrom attacks like UnclearBallot.

Other mitigations. Outside of E2E, there may be other heuristic mitigationsthat can be easily implemented even in deployed voting systems to make ourattack somewhat more difficult. As mentioned above, randomizing candidateorder on each ballot increases the computation required to perform our attack.Voters drawing outside the bubbles can also defeat our attack, though this mightalso result in their votes not counting and may be circumvented by replacing thewhole race on the ballot image with a substituted one. Collecting ballot images

14

from a different source than the tabulator makes our attack more difficult, asvotes now have to be changed in two places. Other standard computer securitytechnologies, like secure file systems, could be used to force the attacker to alterballot images in a way that also circumvents protections like encryption andpermissions.

Detection. Technologies that detect image manipulation may also provide somemitigation. Techniques like those discussed in [3–5,38], among others, could beadapted to try to automatically detect moved marks on ballots. However, asnoted by Farid [19], image manipulation detection is a kind of arms race: given afixed detection algorithm, adversaries can very likely find a way to defeat it. Inour context, an attacker with sufficient access to the voting system to implant amanipulation algorithm would likely also be able to to steal the detector code.The attacker could improve the manipulation algorithm or simply use the detectoras part of their mark-moving calculus: if moving a mark will trip the detector,an attacker can simply opt not to move the mark.

While a fixed and automatic procedure for detecting manipulation can providelittle assurance, it remains possible that an adaptive approach to detection couldbe a useful part of a post-election forensics investigation. However, staying onestep ahead of sophisticated adversaries would require an ongoing research programto advance the state of the art in detection methods.

A less costly and more dependable way to detect ballot manipulation detectionwould be to use a software independent audit trail to confirm election outcomes.This can be accomplished with risk-limiting audits, and the software independenceenabled by RLAs provides other robust security properties to elections, includingdefending against other potential attacks on tabulation equipment and servers.

Future work. We have only focused on simple-majority elections here, becausethose are the kinds of elections used by jurisdictions that do image audits. Auditsof more complex election methods, like instant-runoff voting or D’Hondt, havebeen examined to some extent [36,41], but future work is needed into audits ofthese kinds of elections altogether. Because the marks made in these elections aredifferent than the kind we’ve discussed here, manipulating these ballot imagesmay not be able to employ the same image processing techniques we have used.Additionally it may be difficult for malware to know how many marks it needsto move, since margins in complex elections are difficult to compute. We leaveexploration of image manipulation of these elections to future work.

7 Conclusion

In this paper, we demonstrated an attack that defeats ballot image audits ofthe type performed in some jurisdictions. We presented an implementation usinga real scanner, and evaluated our implementation against a set of real ballotsand a set of systematically marked ballots from a variety of ballot styles. Our

15

attack shows that image audits cannot be relied upon to verify that elections arefree from computer-based interference. Indeed, the only currently known way toverify an election outcome is with direct examination of physical ballots.

Acknowledgements

The authors thank Vaibhav Bafna and Jonathan Yan for assisting in the initialversion of this project. They also thank Josh Franklin, Joe Hall, Maurice Turner,Kevin Skoglund, Jared Marcotte, and Tony Adams for their invaluable feedback.We also thank our anonymous reviewers and our shepherd, Roland Wen. Thismaterial is based upon work supported by the National Science Foundation undergrant CNS-1518888.

References

1. Adida, B., Rivest, R.L.: Scratch and Vote: Self-contained paper-based cryptographicvoting. In: ACM Workshop on Privacy in the Electronic Society. pp. 29–40 (2006)

2. Bajcsy, A., Li-Baboud, Y.S., Brady, M.: Systematic measurement of marginal marktypes on voting ballots. Tech. rep., National Institute for Standards and Technology(2015)

3. Bayar, B., Stamm, M.C.: A deep learning approach to universal image manipulationdetection using a new convolutional layer. In: Proceedings of the 4th ACM Workshopon Information Hiding and Multimedia Security. pp. 5–10. ACM (2016)

4. Bayram, S., Avcıbas, I., Sankur, B., Memon, N.: Image manipulation detection withbinary similarity measures. In: 2005 13th European Signal Processing Conference.pp. 1–4. IEEE (2005)

5. Bayram, S., Avcibas, I., Sankur, B., Memon, N.D.: Image manipulation detection.Journal of Electronic Imaging 15(4), 041102 (2006)

6. Bell, S., Benaloh, J., Byrne, M.D., DeBeauvoir, D., Eakin, B., Fisher, G., Kortum,P., McBurnett, N., Montoya, J., Parker, M., Pereira, O., Stark, P.B., Wallach, D.S.,Winn, M.: STAR-vote: A secure, transparent, auditable, and reliable voting system.USENIX Journal of Election Technology and Systems 1(1) (Aug 2013)

7. Benaloh, J.: Administrative and public verifiability: Can we have both? In:USENIX/ACCURATE Electronic Voting Technology Workshop. EVT ’08 (Aug2008)

8. Benaloh, J., Jones, D., Lazarus, E., Lindeman, M., Stark, P.B.: SOBA: Secrecy-preserving observable ballot-level audit. In: proc. Proc. USENIXAccurate ElectronicVoting Technology Workshop (2011)

9. Bernhard, M., Benaloh, J., Halderman, J.A., Rivest, R.L., Ryan, P.Y., Stark, P.B.,Teague, V., Vora, P.L., Wallach, D.S.: Public evidence from secret ballots. In:International Joint Conference on Electronic Voting. pp. 84–109. Springer (2017)

10. Bowen, D.: Top-to-Bottom Review of voting machines certified for use in California.Tech. rep., California Secretary of State (2007), https://www.sos.ca.gov/elections/voting-systems/oversight/top-bottom-review/

11. Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)12. Carback, R., Chaum, D., Clark, J., Conway, J., Essex, A., Herrnson, P.S., Mayberry,

T., Popoveniuc, S., Rivest, R.L., Shen, E., Sherman, A.T., Vora, P.L.: ScantegrityII municipal election at Takoma Park: The first E2E binding governmental electionwith ballot privacy. In: 18th USENIX Security Symposium (Aug 2010)

16

13. Chung, K.K.t., Dong, V.J., Shi, X.: Electronic voting method for optically scannedballot (Jul 18 2006), US Patent 7,077,313

14. November 6, 2018 general election. https://dochub.clackamas.us/documents/drupal/f4e7f0fb-250a-4992-918d-26c5f726de3c

15. Clear Ballot: ClearAudit, https://clearballot.com/products/clear-audit16. Dominion Voting: Auditmark. https://www.dominionvoting.com/pdf/DD%

20Digital%20Ballot%20AuditMark.pdf17. Dominion Voting: Cambridge Case Study. https://www.dominionvoting.com/field/

cambridge18. Election Integrity Oregon, https://www.electionintegrityoregon.org19. Farid, H.: Digital forensics in a post-truth age. Forensic science international 289,

268–269 (2018)20. Feldman, A.J., Halderman, J.A., Felten, E.W.: Security analysis of the Diebold

AccuVote-TS voting machine. In: USENIX/ACCURATE Electronic Voting Tech-nology Workshop. EVT ’07 (Aug 2007)

21. Hall, J., Miratrix, L., Stark, P., Briones, M., Ginnold, E., Oakley, F., Peaden, M.,Pellerin, G., Stanionis, T., Webber, T.: Implementing risk-limiting post-electionaudits in California. In: 2009 Workshop on Electronic Voting Technology/Workshopon Trustworthy Elections. pp. 19–19. USENIX Association (2009)

22. Ji, T., Kim, E., Srikantan, R., Tsai, A., Cordero, A., Wagner, D.A.: An analysis ofwrite-in marks on optical scan ballots. In: EVT/WOTE (2011)

23. Jones, D.W.: On optical mark-sense scanning. In: Towards Trustworthy Elections,pp. 175–190. Springer (2010)

24. Lindeman, M., Halvorson, M., Smith, P., Garland, L., Addona, V., McCrea,D.: Principles and best practices for post-election audits (Sep 2008), http://electionaudits.org/files/bestpracticesfinal 0.pdf

25. Lindeman, M., Stark, P.: A gentle introduction to risk-limiting audits. IEEE Securityand Privacy 10, 42–49 (2012)

26. Lindeman, M., Stark, P., Yates, V.: BRAVO: Ballot-polling risk-limiting audits toverify outcomes. In: 2011 Electronic Voting Technology Workshop / Workshop onTrustworthy Elections (EVT/WOTE ’12). USENIX (2012)

27. Maryland House of Delegates: House Bill 1278: An act concerning election law– postelection tabulation audit. http://mgaleg.maryland.gov/2018RS/bills/hb/hb1278E.pdf

28. Maryland State Board of Elections: 2016 post-election audit report. http://dlslibrary.state.md.us/publications/JCR/2016/2016 22-23.pdf (12 2016)

29. Maryland State Board of Elections: December 15, 2016 meeting minutes. https://elections.maryland.gov/pdf/minutes/2016 12.pdf (Dec 2016)

30. McDaniel, P., Blaze, M., Vigna, G.: EVEREST: Evaluation and validation ofelection-related equipment, standards and testing. Tech. rep., Ohio Secretary ofState (2007), http://siis.cse.psu.edu/everest.html

31. Mebane, W., Bernhard, M.: Voting technologies, recount methods and votes inWisconsin and Michigan in 2016. 3rd Workshop on Advances in Secure ElectronicVoting 2018 (2018)

32. National Academies of Sciences, Engineering, and Medicine: Securing the Vote:Protecting American Democracy. The National Academies Press, Washing-ton, DC (2018), https://www.nap.edu/catalog/25120/securing-the-vote-protecting-american-democracy

33. National Conference of State Legislatures: Post-election audits (Jan-uary 2019), http://www.ncsl.org/research/elections-and-campaigns/post-election-audits635926066.aspx

17

34. Rivest, R.: On the notion of ‘software independence’ in voting systems. Phil. Trans.R. Soc. A 366(1881), 3759–3767 (October 2008)

35. Ryan, P.Y.A., Bismark, D., Heather, J., Schneider, S., Xia, Z.: Pret a Voter: Avoter-verifiable voting system. IEEE Transactions on Information Forensics andSecurity 4(4), 662–673 (2009)

36. Sarwate, A.D., Checkoway, S., Shacham, H.: Risk-limiting audits and the margin ofvictory in nonplurality elections. Statistics, Politics and Policy 4(1), 29–64 (2013)

37. ScannerOne: Kodak i5600. http://www.scannerone.com/product/KOD-i5600.html38. Stamm, M.C., Liu, K.R.: Forensic detection of image manipulation using statistical

intrinsic fingerprints. IEEE Transactions on Information Forensics and Security5(3), 492–506 (2010)

39. Stark, P.: Conservative statistical post-election audits. Ann. Appl. Stat. 2(2),550–581 (2008)

40. Stark, P.: Super-simple simultaneous single-ballot risk-limiting audits. In: 2010Electronic Voting Technology Workshop / Workshop on Trustworthy Elections(EVT/WOTE ’10). USENIX (2010)

41. Stark, P.B., Teague, V., Essex, A.: Verifiable European elections: Risk-limitingaudits for D’Hondt and its relatives. USENIX Journal of Election Technology andSystems (JETS) 1, 18–39 (2014)

42. Unisyn Voting Solutions: OpenElect OCS Auditor. https://unisynvoting.com/openelect-ocs/

43. U.S. Election Assistance Commission: Certificate of conformance:ClearVote 1.5. https://www.eac.gov/file.aspx?A=zgte4IhsHz%2bswC%2bW4LO6PxIVssxXBebhvZiSd5BGbbs%3d (2019)

44. Verified Voting Foundation: The Verifier: Polling place equipment (2019), https://www.verifiedvoting.org/verifier/

18