NOTE 10: CH. 6: CONTINUOUS TESTING

Copyright 2015, Robert W. Hasker

Reviewing CI Setup

So what do the steps do? Key concept: search path

Windows: for each command, run through PATH to find first folder containing specified command (an executable with the same name)

Variables: strings that can be retrieved from the environment on demand

JDK_PATH, JAVA_HOME: so java computer and runtime knows where libraries are located

Values of variables retrieved from “owner process” when create new ones – variables are inherited

System variables: shared by all users, recorded by OS when it starts

More on CI Setup

DOS prompt: variables referred to by %X% Eg: open DOS prompt and enter the

command

echo %JAVA_PATH% Unix uses $JAVA_PATH, hence some of

the typos Much of the setup has to do with

getting the proper tools in your path (javac, ant, junit libraries)

Windows services: background jobs

Reliability

Systems Engineering: • Linear System: response to

complex input can be described as a sum of responses to simpler inputs• Eg: Wave propagation• Nonlinear examples: AC power

flow, almost any complex system

• Reliability for linear systems: product of reliability of each component

A: 90% Reliabl

e

B: 90%

C: 90%

Reliability: .93 = .73

Reliability in Software Car that works

73% of the time: find a shop or dealer

How many components in software?

How can we achieve high reliability for SW?

Is unit testing enough?

Integration, system, acceptance Performance testing

based on operational profile - given expected types of inputs, ensure can handle load

What can you learn from this type of testing? What can't you learn from this type of testing?

Stress test Answers what happens at limits/high loads what happens if exceed limits

What do you want to happen? Goal: graceful degredation - bend but don't break

More testing methods

Security testing Maybe something Sony should invest in? How would you test security?

A particular problem for websites Database injection, cross-site scripting Brakeman Rails security scanner Fuzz checkers – inserting random data

Wapiti: identifies scripts, forms in which can inject data

Usability Non-functional requirements in general

Must identify testable component!

Testing strategy

CI: run faster tests first Generally, this means running unit tests first System tests: often require setups Likely: run unit tests always, system tests on

demand Standing issue: regression suite can

easily get too large to run all Research question: can we identify a subset to

run routinely, but save full run for release testing?

But is that suggestion contrary to CI?

Qualities of good tests

Thorough – coverage Start with statement coverage

Repeatable Generate small data sets that have

predictable outputs Test runs should not depend on each

other Understandable

Maintainable Easy to identify failed case

Testing

What about “grandma testing”?

Testing

What about “grandma testing”? Sexist? Ageist?

"Local grandmother finally uses printer"

CES 2009: hall for products for retirees

Testing

What about “grandma testing”? Sexist? Ageist?

"Local grandmother finally uses printer"

Is grandma-using-the-computer themodern equivalent of racist jokes?

CES 2009: hall for products for retirees

Testing

What about “grandma testing”? Sexist? Ageist? Fuzz testing tools

Testing

What about “grandma testing”? Sexist? Ageist? Fuzz testing tools

What if need large data sets? Test oracles Version matching

Testing

What about “grandma testing”? Sexist? Ageist? Fuzz testing tools

What if need large data sets? Test oracles Version matching

Key: differentiate between robustness, correctness

Integration Testing

Common errors: interface misuse: client doesn't satisfy it's

obligations eg: not calling an initialization routine before

executing an operation interface misunderstanding: incorrect

assumptions made by developer of client timing errors: data producers and consumers

may operate at different speeds; if this communication is not protected then may try to consume data before it's produced or produce data when consumer not ready for new information

Integration Testing

Guidelines test extreme ends of ranges in calls test passing NULL pointer whenever

possible stress testing to reveal timing problems if components use shared memory, test

operations being invoked in different sequences

System Testing

Focus: functional testing that is, are there errors in features?

In some sense, this is the only test that matters tests all functionality seen by customer why shouldn't we do all testing at this

level? harder to test deeply harder to know how to fix system if test fails

Acceptance testing: system testing w/ focus on common scenarios

UI Testing in Java

Assumption: painful but readily available

Research: lots of vaporware, abandonware Many solutions for web, but not a lot for

Java FrogLogic, RAutomation,

MarathonTesting, UISpec4J, Mspec, abbot Method that works:

JavaWorld TestUtils; see also counter.zip

UI Testing in Java

Each component: call .setName with unique string

Test code: TestUtils.getChildNamed(frame,

name-of-child)

Use .doClick, .value(), etc. to exercise code Robust

Doesn’t depend on screen locations, specialized test frameworks

But no auto-capture/replay

Manual Test ProcedureStep Req Pass Conditions Pass?

1. Select the Biology program.

UIR-2

System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52

P / F

2. Double-click on Class# 1330

UIR-1System includes Class# 1330 in schedule at bottom

P / F

3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01)

UIR-9System displays Class# 1331 with a pink background

P / F

4. UIR-9All sections listed between #1311 and #1331 have a white background

P / F

5. Select the GENENG program.

UIR-2System displays general engineering courses

P / F

6. UIR-9

GENENG 1030 sections 01-07 have a pink backgroundall other sections of GENGENG 1030 have a white background

P / FP / F

7. UIR-9All sections of GENGENG 1000 have a white background

P / F

Coverage

See se3800/samples/triangle.html Coverage from triangle 3 2 1 Working with partner…

Write tests which pass Identify changes to code which pass those

tests but fail to meet specification

What is statement coverage? How does this differ from branch

coverage?

Coverage Decision: Boolean expression Condition: (atomic) component of a Boolean

expression Decision coverage: every Boolean

expression evaluates to both true and false How different from statement coverage?

Condition coverage: every condition evaluates to both true and false Too easy: consider cases for a && b

Multiple condition coverage: all combinations executed How many tests needed for a && b && c?

MCDC Modified Condition Decision Coverage:

Every entry, exit point executed at least once Every decision has taken all possible outcomes at

least once Every condition has taken all possible outcomes

at least once Every condition (in a decision) has been shown to

independently affect that decision’s outcome. With this definition, number of tests is O(N) See

A Practical Tutorial on Modified Condition/Decision Coverage, NASA/TM-2001-210876 for a tutorial

Path Testing

Statement, decision, MCDC: all about executing logic

Ultimate goal: execute every path Enumerate paths for GCD example:

printf("The gcd of %d and %d", x, y); while ( x != y ) { if ( x > y ) x = x - y; else y = y - x; } printf( " is %d.\n", x );

printf("gcd of %d and %d", x, y);while ( x != y ) { if ( x > y ) x = x - y; else y = y - x; } printf( " is %d.\n", x );

Mutation Testing

Change code, run tests Some test should fail

How to generate mutants? That is: what operators?

Offutt, Rothermel, Lee, Untch, and Zapf. Sufficient Mutant Operators, TOSEM, April 1996

Too many mutants!

Number of mutants from 22 operations:

Are the most expensive mutants necessary?

Examined impact of removing each

Result: minimum operator set ABS: insert calls to an absolute value

function AOR: replace all arithmetic ops by

every syntactically legal alternatives LCR: replaces AND, OR by all logical

connectors ROR: replace (modify) relational

operators UOI: insert unary operators

Evaluation

Five operators, responsible for ~17% of all mutants, sufficient for 10 Fortan test programs

In general: get O(Lines + References) mutants

Constant for O is large! Above 10 programs (200 lines): 231,972

mutants Practical for critical routines

Acceptance Tests & APIs How to write acceptance tests for an API?

Our model: acceptance test = story/scenario Issue: an API is not a user!

Solution: Cohn: Writing User Stories for Back-end Systems Personify subsystems Epic: “As a bank, I want to receive a file showing all

checks to be cleared so that I can debit and credit the right accounts.”

“As a bank, I want to receive a 5300 file with correctly formatted single-line deposit entry records so that I can process them.”

Write test to the resulting story.

Testing Review

Unit, Integration, System, Acceptance

Coverage: making sure all code executed

Mutation testing: testing your tests Acceptance testing and APIs

What next?

How to know when we’re done? Exercise: research network

Methods & Tools: No risk, no need to test! Done depends on identified risks:

minimal criterion is that you are done testing when risks are reduced to an acceptable level

Is there an acceptable level of risk for safety-critical software?

Goal-directed design

What do you want to know about user interface design?