TGDC Meeting, December 2011 Michael Kass National Institute of Standards and Technology Update on...

TGDC Meeting, December 2011

Michael KassNational Institute of Standards and

Technologyhttp://vote.nist.gov

Update on SAMATE Automated Source Code Conformance Verification Against VVSG 1.0 Requirements


Outline Progress report on automated source

code conformance verification effort introduced at last TGDC meeting

Lessons learned Next steps


Software Assurance Metrics and Tool Evaluation (SAMATE)

NIST/DHS co-sponsored project Goal - measure the effectiveness of software

assurance tools to identify weaknesses and vulnerabilities in software

Asked by EAC to assist voting system manufacturers and test labs improve a (heavily manual) source code review process by developing a uniform automated conformance verification capability for voting system source code against VVSG 1.0 software requirements


Tool Automation Progress Finished and tested automation of VVSG 1.0 rules for Java (still documenting C/C++

work) 50 custom rules written for the “Checkstyle” open source Java code analysis tool 41 verification tests (sample “bad” code) against those rules, to verify tool correctness against

requirements Ran customized tool against 1M+ lines of Java source code Bundled Java tool rules and tests into ZIP and tar archive files for distribution to TGDC, VSTLs

and manufacturers

[1] "National Information Assurance Glossary"; CNSS Instruction No. 4009 National Information Assurance Glossary


Requirement Automation Overview

Identified 58 software requirements in VVSG (since last TGDC report) 26 requirements fully/partially automated for Java 12 requirements do not apply to Java language 14 requirements require entirely human analysis 3 requirements could be automated, but require

“custom” knowledge about code or documentation 2 requirements could be verified with a “more capable”

tool 1 requirement is ambiguous (not automatable)


Findings: Requirement Interpretation Issues

We identified 10 requirement interpretation issues that require clarification The term “module” is often used in VVSG 1.0 without

reference to type (method/function, class, library) What constitutes “intentional exception” is not defined Logical nesting limit requirement needs clearer definition Scoping clarification needed for names that differ by only

one character Module line count requirement (type of module, and which

lines “count”) not defined (we assumed module=method)


Findings: Requirement Interpretation Issues

Additional requirement clarification needed for: Indentation requirements not specific (“clear” and

“consistent” is all the requirement specifies) Scoping clarification needed for “unique” name comparison

in code Functional, test and branch statements not clearly defined Definition of what a “library module” is for Java is not

specified Lowest shared access for a resource is not clearly defined


Findings: Even “Clear” Requirements May Be Interpreted Differently

An example: “Initializes every variable upon declaration where permitted” (VVSG 1.0 2:5.4.2) int varOne=0; // VVSG conformant static int varTwo; // non-VVSG-conformant

What if Java automatically initializes varTwo to a value of 0 for you? In some cases, this happens at runtime

Some manufacturers believe that the VVSG requirement is satisfied without explicitly assigning a value to a variable at declaration, and have written their code that way


Findings: Tool Automation Requires An Unambiguous Specification

Need mutual understanding of VVSG requirement meaning among voting system manufacturers, VSTLS and the EAC

The “devil is in the details” to unambiguously specify requirements for source code

We observed some “disconnects” between voting system code and SAMATE interpretation of VVSG requirements (from our July 2011 visit to Wyle Labs)


Findings: Where Possible, SAMATE Should

Customize Tools Manufacturers are Already Using

Where voting system manufacturers are already using automated tools, SAMATE should (first) try to customize the tools that manufacturers are using Two manufacturers currently using “Checkstyle” tool for

Java code conformance verification SAMATE rules can complement the Checkstyle rules the

manufacturer already uses without “adding another tool” This requires a dialog between SAMATE and

manufacturers to identify which tools they are using


Findings: Need to Maximize SAMATE Resources

To make best use of our resources, SAMATE needs to know which tools, programming languages, compilers, libraries and coding conventions manufacturers are using

70% of voting system source code is written in C/C++, Java and C# languages (from discussion with VSTLs)

Manufacturer’s compiler/library selection also dictate which tools SAMATE chooses for automation verification (e.g. GNU vs. Microsoft C++)

If manufacturers are using industry coding conventions (not VVSG 1.0 requirements), then SAMATE should focus its automated verification efforts against those industry coding conventions

Synchronize SAMATE work with future VSTL testing work


Next Steps TGDC review of customized Java tool and tests

Discuss any issues with the SAMATE work Resolve those issues

Distribute SAMATE tool rules and tests to manufacturers and VSTLs

Two manufacturers are using the Checkstyle tool now (using their own custom rules)

Get manufacturer/VSTL feedback on their interpretation and implementation of VVSG requirements in their tools

Resolve those differences (via RFI or other means) Incorporate changes into future releases of VVSG as appropriate Publish a final distribution of customized tools and tests


Next Steps Have a discussion with voting system

manufacturers and VSTLs to determine where SAMATE should focus its future efforts in: Tools Programming languages and compilers Coding conventions Future VSTL work (new systems, re-certifications)


Next Steps Investigate how SAMATE can contribute to

automated conformance verification against VVSG 1.1 software integrity requirements Most VVSG 1.0 “style” requirements are removed in

VVSG 1.1 (in lieu of using industry-accepted coding style conventions)

VVSG 1.1 requirements are more “integrity focused” Not trivial to find integrity bugs in source code Requires more sophisticated tools than “style checkers” Requirements include identifying race conditions, deadlocks,

livelocks, pointer validation, dynamic memory allocation, numeric overflows and CPU traps


Next Steps Integrate SAMATE VVSG work with SAMATE “Common

Weakness Enumeration (CWE) Compatibility Effectiveness” program for testing automated vulnerability analysis tools

CWE is a unified, measurable set of software weaknesses defined in an online dictionary sponsored by DHS and implemented by MITRE

Tools are assessed against their ability to identify CWEs in software with known weaknesses and known source code “complexities” (that can influence a tool’s ability to correctly identify a weakness)

SAMATE can leverage this work in developing test suites to verify tool effectiveness against VVSG source code integrity requirements


Summary SAMATE was successful at fully or partially automating

Java source code verification against VVSG 1.0 requirements that could be automated

Need to resolve semantic definitions of some VVSG 1.0 software requirements (with manufacturers, labs and EAC) for uniform automated conformance verification uniformly across all tools

Need to prioritize SAMATE resources against voting system manufacturer’s tools, languages, compilers and coding conventions to maximize impact for future work

Need to look forward to VVSG 1.1 software integrity requirements, and how SAMATE can assist manufacturers and VSTLs through tool effectiveness testing


Discussion

TGDC Meeting, December 2011 Michael Kass National Institute of Standards and Technology Update on...

Documents

Transcript of TGDC Meeting, December 2011 Michael Kass National Institute of Standards and Technology Update on...