© Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005...

TTEEMM CCoonnssuullttiinngg,, LLPP

© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB

Considerations In Designing a

Conformity Assessment System for Voting Equipment

Presented atThe Voting Systems Testing Summit

November 28-29, 2005Sacramento, CA



Purpose of Presentation

This presentation discusses:

• Elements common to conformity assessment systems

• The EAC system for certification and decertification of voting equipment

• Considerations for state and local programs

• Potential for unintended consequences



Certification of a product is a means of providing assurance that it complies with specified standards and other normative documents.

Several types of certification systems exist and are under review for the EAC system:

•Some comprise type testing only

•Other systems include initial testing and field surveillance

•Still others include initial testing of a product and assessment of its suppliers’ quality systems, followed by routine audits that take into account the factory quality system and the testing of samples from the factory and the open market.



Components of a Certification System

•Initial Type Testing

•Assessment of the Supplier’s Quality System

•Field Information & Feedback

•User Involvement



Key Participants

•EAC

•State certification authorities

•Testing Laboratories

•Vendors

•State & local election officials



Key Issues for Certification System

•What is a minimal acceptable system?

•Are the testing lab/testers/lab assessors qualified?

•Will the vendor deliver units within manufacturing tolerances to those tested?

•How will the election officials know if non-compliant units are delivered and what corrective actions can it take?

•Will election officials and poll workers use the systems as intended?



Primary ConsiderationNational Program The primary concern of the EAC

program is that the system design, as represented in the system submitted for testing, meets minimum requirements.

State Program State certification seeks to assure that:

1. Systems meet specific state requirements.

2. May be used as that state administers elections.

3. The vendor can give reasonable assurance that the units delivered meet and continue to meet the requirements over their useful life.



Primary ConsiderationPurchase Evaluation Which system offers the best value

among systems offered.

Receiving Inspection Are the delivered systems functional and within manufacturing tolerance of those certified by the EAC and state officials?

Pre & Post Election Are the systems functioning correctly and unaltered from their certified condition?



Coordinated Processes

Processes are necessary to implement the components of a conformity assessment system for voting systems and to provide reasonable assurance that the key concerns of such a system are carefully addressed.

The best confidence will be gained if the various authorities involved in certifying and selecting voting equipment coordinate their efforts so as to maximize the value contributed from each function.



EAC / NISTTGDC

Technical Guidelines

EAC / NISTTGDC

Technical GuidelinesCandidate System

Accredited

Test Laboratory

Accredited

Test Laboratory EACEAC Technical Review

Team

Technical Review

Team

Certified SystemSystem

Description

System

DescriptionNIST

Software Ref. Lib.

NIST

Software Ref. Lib.

Vendor

Quality & ConfigurationControl Processes

Vendor


State

Certification

State

Certification



Certified SystemSystem

Description

System

DescriptionNIST

Software Ref. Lib.

NIST

Software Ref. Lib.

Vendor


Vendor


State

Certification

State

Certification

Field Deploymentand Use

Field Deploymentand Use

Purchase SelectionPurchase Selection

Purchase SelectionPurchase Selection

Pre & Post Election

Testing

Pre & Post Election

Testing



EAC / NISTTGDC


EAC / NISTTGDC


Accredited

Test Laboratory

Accredited

Test Laboratory

EACEACTechnical Review

Team

Technical Review

Team

NIST

NVLAP&

Software Ref. Lib.

NIST

NVLAP&

Software Ref. Lib.

VendorsVendors

State & LocalElection Officials

State & LocalElection Officials

CitizensCitizensLocal

Receiving Inspectors

Local

Receiving Inspectors



System Description

The ability to determine that a system is identical to that considered at other steps in the process is critical to the validity of the conformity assessment system.



Typical FCC Submission

The FCC routinely requires external AND internal photos accompany all submissions.

For Software Controlled Radios source code is required for an equipment grant.




External Photos




Internal Photos



Challenge for Vendors

Is there a way by which high confidence many be delivered that a system is exactly the same without

also transmitting proprietary information?



Considerations

How does each step of the process assist the next?

Could the ITA’s include a section in each test report highlight information and observations that would be helpful to state certification personnel?

Could state certifiers transmit observations to those making purchasing decisions?



Considerations

What is the repeatability of these evaluations?

How often will the same system receive the same evaluation if submitted at different time or to different ITA’s?



Considerations

How does field performance and field experience information come back into the system to improve it?

How will field experience come into the system to improve EAC and state certification processes?



Unintended Consequences

Cost of certification will limit the number of systems offered and may eliminate vendors.

With more rigorous testing and evaluation vendors will pay for fewer systems to be certified.

Vendors will bring new systems to market less often.

Some vendors may abandon this business.




Will the time and expense of certifying changes and upgrades delay good and needed changes?

What is the right balance between careful evaluation of all changes and not making these evaluations so costly that needed improvements are delayed or not introduced at all?




How will the necessary coordination of administrative processes and equipment requirements be maintained and evaluated?

Election management and equipment requirements must work together. How will this be assured? Where several alternative solutions are possible how will the linkage be maintained between equipment requirements and election management practices?



ProposalsDocuments are needed that define the interaction and expectations of each participant in the process.

Direct communications from each step in the process to succeeding steps may be very helpful.

Clarification is needed as to what is not done in preceding steps in the process.



Conclusion

A well constructed certification system provides satisfactory answers to central issues:

•What is a minimal acceptable system?

•Are the testing lab/testers/lab assessors qualified?

•Will the vendor deliver units within manufacturing tolerances to those tested?

•Will the election officials know if non-compliant units are delivered and what actions can they take?

•Will election officials and poll workers use the systems as intended?

© Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005...

Documents

Transcript of © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005...