Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and...

Test Specifications

A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution

Greg Cooksey

Test Specifications-2-

Talk Outline

• Test suite challenges and basis for our new architecture

• The specification system• A walk through the new Dyninst test suite architecture

• Other features, status notes, and future plans


Idealized Dyninst Test

• Mutator program– Uses Dyninst to analyze or modify mutatee

• Mutatee program– Simply verify that modification occurred

• Output– passed, failed, or skipped

Simple, right?

mutator

Dyninst

mutatee

outputtest


Not That Simple• Multiple platforms

– Architecture– operating system– operating system version

– 32 vs. 64 bit

• Multiple compilers– Native– GNU– Other

• Multiple languages– C– C++– Fortran

• Multiple build parameters– Optimization levels– Stripped vs. unstripped

• Multiple test parameters– Create vs. attach

• Multiple mutator-mutatee mappings

And not all combinations are valid


Our Goal• Write simple test cases:

• Modify specification file– Don’t modify code when adding a test

• Automatically do the rest– Build the test programs, with appropriate variations and parameters

– Run the test with appropriate parameters– Collect results and insert into database

mutator() {

insertSnippet(…)

. . .

}

foo() {

. . .

}


New Infrastructure• Specification language and compiler

– Specifications for almost all aspects of test suite

• Test suite build system– Standard build system– Makefiles are output of spec. compiler

• Test driver– Tests are plugins– List of tests to run is output of spec. compiler

• Results parser and databaseUser only needs to write test cases and test specifications


Specifications• What?

– Platforms• Architecture• Operating system• File name conventions• Compiler availability

– Compilers• Language• Standard flags• Option-parameter string mapping

– Languages– Test cases

• Mutator sources• Mutatee sources• Mutator-mutatee mappings• Build configurations• Run configurations

– Test suite glue• Standard libraries• Constraints

• How?– We use simple tuples, written as Prolog clauses

• Prolog lets us naturally express relationships between tuples


Test Suite Architecture

Build Stage

Specifications

Run Stage Data Collection

Test sources



Build Stage Run Stage Data Collection

spec file spec compiler

Test sources


Specification Compiler

mutator spec

mutatee spec

test spec

standard libs

compiler spec

Prolog

Specifications

mutator tuple

mutatee tuple

test tuple

Tuples

Python

makefile

test lists

Scripts

test suite glue


mutator(‘example’, [‘example.C’]).

mutatee(‘example’, [‘example_mutatee.c’]).

compiler_for_mutatee(‘example’, Compiler) :-

compiler_language(Compiler, ‘c’);

compiler_language(Compiler, ‘c++’).

mutator_mutatee(‘example’, ‘example’).

test_runs_everywhere(‘example’).

test_runmode(‘example’, ‘both’).

test_start_state(‘example’, ‘stopped’).

Test Specificationmutator source

mutatee source

valid compilers

mutator-mutatee mapping

build and run constraints



Build Stage Run Stage Data Collection


Test sources




mutator compiler

mutatee compilers

build scripts (makefiles)


Test sources


Build scripts

• Specification compiler generates makefile

• Mutator built as shared library plugin– example.so

• Multiple mutatees built with build parameters encoded into names– example.mutatee_g++_none– example.mutatee_g++_low– . . .




mutator compiler

mutatee compilers



Test sources



Data Collection

mutator compiler

mutatee compilers


spec file spec compilerrun scripts (test lists)

test driver src Dyninst compiler

test driver

mutator .so

mutatee1

mutatee2

mutatee3

Test sources


Run scripts

• Prolog generates tests as tuples– (example, example.mutatee_gcc_low, createProcess, stopped)

• Python script creates lists of tests that are compiled into the test driver

• Test driver executes lists of tests and produces results output



Data Collection

mutator compiler

mutatee compilers




test driver

mutator .so

mutatee1

mutatee2

mutatee3

Test sources



mutator compiler

mutatee compilers




test driver

mutator .so

mutatee1

mutatee2

mutatee3output

logparser

database

Test sources


Collecting Output

• Test output“example –mutatee example.mutatee_g++_low”

All tests passed

• Output is human readable• Added to database

– (example, g++, low, createProcess): passed


Flexible Testing Structures

• Build each test as separate binary– High isolation, slow performance

• Build several tests into a single binary– Balance test isolation vs. performance


Not Just Tests

• Specifications also used for– Platforms

•Specify architecture, operating system, filename conventions, compiler availability*

– Compilers•Specify languages*, option - command line parameter mapping, standard flags, platform availability*

– Languages•Specify filename conventions, compilers*


Example Platform

% Linux 2.6 i386 specification

% paragraph

platform(‘i386’, ‘linux’,‘linux2.6’,

‘i386-unknown-linux2.6’).

compiler_platform(‘gcc’, ‘i386-unknown-linux2.6’).

object_suffix(‘i386-unknown-linux2.6’, ‘.o’).

library_prefix(‘i386-unknown-linux2.6’, ‘lib’).

library_suffix(‘i386-unknown-linux2.6’, ‘.so’).


Example Compiler

% g++ specification paragraph

comp_lang(‘g++’, ‘c++’).

compiler_platform(‘g++’, Platform) :-

platform(_, OS, _, Platform), OS \= ‘windows’.

compiler_optimization_translate(‘g++’, ‘high’, ‘-O2’).

comp_std_flags(‘g++’, ‘$(CXXFLAGS)’).

comp_mutatee_flags(‘g++’, ‘$(MUTATEE_CXXFLAGS_GNU)’).

mutatee_link_options(‘g++’, ‘$(MUTATEE_LDFLAGS_GNU)’).


Status

• Infrastructure functional• Converted 2/3 of old tests; working on the rest

• Will add SymtabAPI tests to new infrastructure

• Tech. report in early summer• Future: integrate into NMI Build & Test system

Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and...

Documents

Transcript of Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and...