CMPSC 311- Introduction to Systems Programming Module: Testing · 2016-12-09 · CMPSC 311 -...

CMPSC 311 - Introduction to Systems Programming

CMPSC 311- Introduction to Systems Programming

Module: Testing

Professor Patrick McDanielFall 2016

CMPSC 311 - Introduction to Systems Programming Page

NASA Mariner 1• 1962 – the Mariner 1

spacecraft was set to do a flyby of Venus‣ Guidance system failed

shortly after takeoff

‣ Destroyed just under 300 seconds from takeoff

‣ Specification error

¯Rn ! Rn


Why testing ….• When developing software you construct logic to

perform some set of often complex tasks‣ Software bugs happen just as a consequence of process

‣ Identifying the bugs before they cause harm is key to being a good programmer (and viability of the company)

‣ Need to assess the “state” of the code• Figure out what parts of the code need work

• Figure out if changes to code cause problems to arise


Validation vs. Verification• Validation is where you run a

program to see if it runs correctly on some input (testing …)

• Verification is used to verify that a program adheres to some invariant (property or behavior) using a proof based system‣ Most non-trivial properties for real

program are undecidable

‣ See CMPSC461 … (very cool!)


Kinds of bugs …• Software may fail for lots of reasons …‣ Specification errors (Mariner 1)

‣ Assumption errors

‣ Logic errors

‣ Code errors

‣ Run-time errorsif (cart = 1) {

printf(”Got a one!”);} else {

printf(”Got a non-one!”);}

¯Rn ! Rn


Software engineering ….• Every software shop has a process for developing code,

which is defined by the software development model‣ E.g., Waterfall model

Requirements Analysis

Coding

Design

Testing (verification)

Maintenance

Release

Alpha vs. Beta testing:Alpha testing


Testing 101• The ability to test depends on a clear and unambiguous

specification (requirements document)‣ Testing with respect to function

‣ If not clear, than untestable

• Note: Industry average:‣ 30-85 errors per 1000 lines of code (LOC)

‣ 0.5-3 errors per 1000 LOC getting through release


Types of testing …• Offline ”code” evaluation strategies‣ Code review inspections

‣ Code analysis

• Online testing‣ Workload testing

‣ Black box vs. white box testing

‣ Regression (maintenance, test changes)

• Testing and the phases of the development lifecycle‣ Unit testing validates modules individually

‣ Alpha testing is “acceptance testing” in house

‣ Beta testing typically with external users

Unit Testing

Beta Testing

Alpha Testing


Lint (splint on Linux)• Lint uses static analysis of code to check for anything

that may be a problem with code‣ Often creating warnings that are not really problems

‣ This is a great way to really cleanse your code, but may take a while and sometimes not worth the trouble to fix all warnings.


Unit testing• Unit testing is a technique where you create code that

exercises the logic and parameters for some module (typically not the entire program)‣ This allows you to create code that validates an

implementation independently from the rest of the program.

‣ Generally, the structure of the code is a single function:• (1) Initialize the module

• (2) Create a loop (of some length)‣ (2a) Exercise each function and validate the return values

• (3) Clean up data

• Note: this approach is used for both functional and performance testing of the module


Unit Testing Example (cache)initialize the cache for a given size

Set of cache items = (create N of these, mark all as not in cache)

loop some number of FIXED_ITERATIONS {

select random operation (read/insert from cache)

pick random cache item X

read:

if X marked as in cache

if read from cache says there good, otherwise FAIL

else

if read from cache says there FAIL, otherwise good

insert:

if X NOT marked as in cache

if insert cache success good, otherwise FAIL

walk cache looking for ejected entries, update cached marks

}

close the cache, freeing all of the elements


Coverage• Coverage is a measure of how much of a program you

have tested‣ Code coverage – how many lines have do tests exercise

‣ Branch coverage – how many branches do tests exercise

‣ Path coverage – how many paths have do test exercise

Note: we are not counting lines that are non-executable (braces, else, …)


Coverage• Code coverage is the amount of code that is executed

when a particular set of inputs is given.• What is the coverage of the following:‣ a=6, b=9 (5/7 = 71%)

‣ a=4, b=5 (3/7 = 42%)

‣ Can you get 100% witha single test case?

int func(int a, int b) {if (a&2) {

b = a<<1;if (b&0x8) {

printf("One");}

} else {b = a^7;

}return(a*b);

}

Note: we are not counting lines that are non-executable (braces, else, …)


Test vectors• Test vectors are carefully selected sets of inputs that

get maximal coverage.

• This can be calculated directly from the code (often)‣ (a&2)&(b&8), !(a&2) == 100% coverage

• When constructing tests you want to generate as few test cases as possible that give you the most coverage


b = a<<1;if (b&0x8) {

printf("One");}

} else {b = a^7;

}return(a*b);

}


Control flow graphs• A control flow graph is a graph of

the path through the code from entry to (any) exit‣ Path coverage means all paths traversed

‣ However, paths are exponential in the number of branches (worst case)

‣ Meaning intractable in many cases …

START

a&2

b = a << 1; b = a^7;

a&b&0x1000

b = a << 1;

END


b = a<<1;if (b&0x8) {

printf("One");}

} else {b = a^7;

}return(a*b);

}


Code reviews• In a professional development organization, you will be

periodically subjected to code review‣ Meeting ”chalk talk”

review process

‣ Offline “expert” review and commentary


Lets get real …• Testing is one of the most

important skills you need to become a professional programmer.‣ Identify bugs by reading your code‣ Learn to use unit testing

‣ Create test vectors to test key parts of your code

‣ Take your time—it will pay off in the end

CMPSC 311- Introduction to Systems Programming Module: Testing · 2016-12-09 · CMPSC 311 -...

Documents

Transcript of CMPSC 311- Introduction to Systems Programming Module: Testing · 2016-12-09 · CMPSC 311 -...