02-Software Testing Introduction

8/2/2019 02-Software Testing Introduction

1/31

Software Testing

Introduction

Software Testing 2

The hardest task Can write a program in this 50 minute class Can also have 50 million possible combinations to

test. Labor involved can cost fortunes. Testing is often the first thing to slip when resources

stretch - but it's the entire organization that sufferswhen a defect causes problems.

Measurements show that a defect discovered duringdesign that costs $1 to rectify will cost $1,000 torepair in production. This is a tremendous (ln) costdifferential and clearly points out the advantage ofearly error detection.


2/31

Software Testing 3

Outline Introduction to the subject

What is... error/bug/fault/failure/testing? Psychology, goals and testing theory Overview of the testing process concepts and

dimensions Testing principles Standard testing questions

Classifying aspects of testing

Types Levels

Software Testing 4

What is...

error: something wrong in software itselffault: manifestation of an error

(observable in software behavior)

failure: something wrong in softwarebehavior

(deviates from requirements)

requirements:for input i,

give output 2*(i^3)

(so 6 yields 432)

software:i=input(STDIN);

i=double(i);i=power(i,3);output(STDOUT,i);

output (verbose):input: 6

doubling input..computing power..output: 1728

bug

error fault

fault + failure


3/31

Software Testing 5

QA Vs QC Your ideas ???

Software Testing 6

The Psychology of Testing

Testing is the process of demonstrating(chng minh, gii thch) that an errors arenot present

Testing is the process of establishingconfidence that the program does what itintends to do

Testing is the process of executingprograms with the intent of finding errors Successful (positive) test: exposes an error


4/31

Software Testing 7

Possible Goals of Testing

Find faults Glenford Myers, The Art of Software

Testing

Provide confidence of reliability (s tin cy) of (probable) correctness (tnh ng n,

chnh xc)

of detection (therefore absence) ofparticular faults (pht hin li c th)

Software Testing 8

What if testing does not detect anyerrors

Either the software is high quality the testing process is low quality

Metrics are required on our testingprocess if we are to tell which is theright answer.


5/31

Software Testing 9

Three themes prevail:

(3 ch ph bin) Quality Assessment (What percentage

of defects are captured by our testingprocess, how many remain?)

Risk Management (What is the riskrelated to remaining defects?)

Test Process Improvement (How longdoes our testing process take?)

Software Testing 10

What is...

testing:by experiment, find errors in software (Myers, 1979) establish quality of software (Hetzel,

1988)a succesful test: finds at least one error gives quality judgment (phn quyt)

with maximum confidence with minimumeffort


6/31

Software Testing 11

Testing Theory (such as it is) Plenty of negative results Nothing guarantees correctness Statistical confidence is prohibitively

expensive Being systematic may not improve fault

detection as compared to simple random testing

So what did you expect, decisionprocedures for undecidable problems?

Software Testing 12

Whats been said? Dijkstra:

Testing can show the presence of bugs, but not the absence Beizer:

1st law: Every method you use to prevent or find bugs leaves aresidue of subtler bugs (xt li nhng li khng d phthin)

2nd law: Software complexity grows to the limits of our abilityto manage it

Beizer:Testers are not better at test design than programmers are at

code design ... Lets start with a global look at the testing process


7/31

Software Testing 13

Concept map of the testing

process

Software Testing 14

Dimensions of software testing1. What is tested?

a. Software characteristics (design/embedded?/language/...)b. Requirements (what property/quality do we want from the software)c. Purpose (what development phase, what kind of errors, what next step)

2. How to test?a. Method (adequacy criterion: how to generate how many tests)b. Assumptions (limitations, simplifications, heuristics)c. Organization (documentation, implementation, execution:platform,

interactive, batch)

d. Who tests? (programmer, user, third party)3. How are the results evaluated?


8/31

Software Testing 15

Dimensions + concept map

1a

1b

1c2a2b

2c

2d

3

Software Testing 16

What information can we exploit(khai thc) ? 1b. Requirements

functional: the behavior of the system is correct precise

non-functional: performance, reliability, compatibility,

robustness (stress/volume/recovery),usability, ...

possibly quantifiable (c th xc nh slng hp l), always vague (lun mh)


9/31

Software Testing 17

What other information can weexploit?

Specifications (formal or informal) in Oracles for Selection, Generation, Adequacy

Designs

Code for Selection, Generation, Adequacy

Usage (historical or models) Organization experience

Software Testing 18

1c. Test purposeSoftware development phases (V-

model):

implementationcode

detaileddesign

specification

requirements acceptancetest

systemtest

integrationtest

unittest


10/31

Software Testing 19

1c. Test purposeWhat errors to find? typical typos functional mistakes unimplemented required features

software does not do all it should do implemented non-required features

software does things it should not do

Software Testing 20

2a. Test method dimensions

data-based

structure-based

black box white boxerror seeding

typical errorsefficiency

...


11/31

Software Testing 21

2b. Assumptions, limitations single/multiple fault:

clustering/dependency of errors testability:

can software be tested? time, resources, accessibility,

perfect repair ...

Software Testing 22

2c. Test organization Documentation

for reuse! Implementation

platform batch? inputs, coordination, ...

Execution

duration, timing, interruptions manual/interactive or automated in parallel on several systems


12/31

Software Testing 23

Testing Principles

Test case must include the definition ofexpected results

A program should not test her/his owncode

A programming organization should nottest its own programs

Thoroughly inspect (xem xt k lng)the results of each test

Test cases must be also written for invalidinputs

Software Testing 24

Testing Principles Check that programs do not do

unexpected things Test cases should not be thrown a way Do not plan testing assuming that there

are no errors The probably of error in a piece of code is

proportional(tng ng) to the errorsfound so far in this part of the code


13/31

Software Testing 25

Test Groups Pros Only people to use a system heavily is

experts; Independent testing is typically more efficient

at detecting defects related to special cases,interaction between modules, and systemlevel usability and performance problems

Programmers are neither trained, normotivated to test

Overall, more of the defects in the productwill likely be detected.

Test groups can provide insight into thereliability of the software before it is actuallyshipped

Software Testing 26

Developers Mindset

A good developer is clever (thng minh) Emphasis on depth of expertise

Product

Expertise


14/31

Software Testing 27

Testers Mindset

A good tester is mischievous (tinh qui)

Emphasis on breadth of expertise

Product

Expertise

Software Testing 28

Slogan !


15/31

Software Testing 29

Testing is in Your Head

Technical

Knowledge

Specifications

Product

Ah! Problem!

Problem

Coverage

ProblemReport

Communication

The important parts of testing dont

take place in the computer or on your desk.

CriticalThinking

DomainKnowledge

Experience

Software Testing 30

A Testers Attitude Cautious (cn thn)

Jump to conjectures (phng on), not conclusions (ktlun).

Practice admitting I dont know. Have someone check your work.

Curious (t m) What would happen if? How does that work? Why did that happen?

Critical (ch trch, ph phn) Proceed by conjecture and refutation (bc b). Actively seek counter-evidence.

Good testersare hard to fool.

(and Courageous)


16/31

Software Testing 31

Standard Testing Questions

Did this test execution succeed or fail? Oracles

How shall we select test cases? Selection; generation

How do we know when weve testedenough? Adequacy

What do we know when were done? Assessment

Software Testing

Classifying aspects of testing


17/31

Software Testing 33

Basic Methods of Testing Black Box White Box

Software Testing 34

Black Box Testing

View the program as a black box Exhaustive testing is infeasible

(khng th thc hin) even for tinyprograms Can never guarantee correctness Fundamental question is economics Maximize investment Example: Partition test


18/31

Software Testing 35

Black Box Tests

Targeted at the apparent simplicity of the software Makes assumptions about implementation

Good for testing component interactions Tests the interfaces and behavior

Input Output

Software Testing 36

White Box Testing

Investigate the internal structure ofthe program

Exhaustive path testing is infeasible Does not even guarantee

correctness Specification is needed Missing paths Data dependent paths

if (a-b < epsilon)


19/31

Software Testing 37

White Box Tests

Targeted at the underlying complexity of the software Intimate knowledge of implementation

Good for testing individual functions Tests the implementation and design

Input Output

Software Testing 38

Equivalence partitioning

Boundary-value analysis

Cause-effect graphing

Error guessing

State space exploration

Black box White box

Coverage/Adequacy

Data flow analysis

Cleanness

Correctness

Mutation

Slicing

Black and White Box TestingMethods


20/312

Software Testing 39

Structural Coverage Testing (In)adequacy criteria

If significant parts of program structure are nottested, testing is surely inadequate

Control flow coverage criteria Statement (node, basic block) coverage Branch (edge) and condition coverage Data flow (syntactic dependency) coverage Various control-flow criteria

Attempted compromise between theimpossible and the inadequate

Software Testing 40

Testing LevelsDifferent testing levels

Unit test (procedure boundaries) Module test Function test (discrepancies external spec.) System test (discrepancies with original objectives) Facility test Volume test

Usability test Security test Performance test ...


21/312

Software Testing 41

Process-Based Reliability

Testing Rather than relying only on properties

of the program, we may use historicalcharacteristics of the developmentprocess

Reliability growth models (Musa,Littlewood, et al) project reliabilitybased on experience with the current

system and previous similar systems

Software Testing 42

Fault-based testing

Given a fault model hypothesized set of deviations from correct

program typically, simple syntactic mutations; relies on

coupling of simple faults with complex faults

Coverage criterion: Test set should beadequate to reveal (all, or x%) faults

generated by the model similar to hardware test coverage


22/312

Software Testing 43

The Budget Coverage Criterion A common answer to when is testing

done When the money is used up When the deadline is reached

This is sometimes a rational approach! Implication 1: Test selection is more important

than stopping criteria per se. Implication 2: Practical comparison of

approaches must consider the cost of test caseselection

Software Testing 44

The Importance of Oracles

Much testing research has concentrated onadequacy, and ignored oracles

Much testing practice has relied on theeyeball oracle Expensive, especially for regression testing makes large numbers of tests infeasible

Not dependable Automated oracles are essential to cost-

effective testing


23/312

Software Testing 45

Sources of Oracles Specifications

sufficiently formal (e.g., Z spec) but possibly incomplete (e.g., assertions in Anna,

ADL, APP, Nana)

Design models treated as specifications, as in protocol

conformance testing

Prior runs (capture/replay)

especially important for regression testing andGUIs; hard problem is parameterization

Software Testing 46

What can be automated? Oracles

assertions; replay; from some specifications

Selection (Generation) YOU scripting; specification-driven; replay variations selective regression test

Coverage (XATAC)

statement, branch, dependence Management (GATE)


24/312

Software Testing 47

Design for Test: 3 Principles Observability

Providing the right interfaces to observe thebehavior of an individual unit or subsystem

Controllability Providing interfaces to force behaviors of

interest

Partitioning Separating control and observation of one

component from details of others

Test Cycle:What You Do With a Build1. Receive the product. Formal builds Informal builds Save old builds.

2. Clean your system. Completely uninstall earlier builds.

3. Verify testability. Smoke testing Suspend test cycle if the product is

untestable.


25/312

Test Cycle:

What You Do With a Build4. Determine what is new or changed.

Change log

5. Determine what has been fixed. Bug tracking system

6. Test fixes. Many fixes fail! Also test nearby functionality.

7. Test new or changed areas. Exploratory testing.

Test Cycle:What You Do With a Build

8. Perform regression testing. Not performed for an incremental cycle. Automated vs. manual Important tests first!

9. Report results. Coverage Observations Bug status (new, existing, reopened, closed) Assessment of quality Assessment of testability


26/312

General Test Techniques

Function testing Domain testing Stress testing Flow testing Scenario testing User testing Regression testing Risk testing Claims testing Random Testing

For any one of these techniques,

theres somebody, somewhere,

who believes that is the

only way to test.

Function Testing

March through the functions

- A function is something the product can do.

- Identify each function and sub-function.

- Determine how you would know if they worked.

- Test each function, one at a time.

Key Idea:

Summary:

Good for: assessing capability rather than reliability


27/312

Domain Testing

Divide the data, and conquer

- A domain is a set of test data.

- Identify each domain of input and output.

- Analyze limits and properties of each domain.

- Identify combinations of domains to test.

- Select coverage strategy:

e.g., exhaustive, boundaries, best representative

Key Idea:

Summary:

Good for: all purposes

Stress Testing

Overwhelm the product

- Select test items and functions to stress.

- Identify data and platform elements that relateto them.

- Select or generate challenging data andplatform configurations to test with:

e.g., large or complex data structures,

high loads, long test runs, many test cases

Key Idea:

Summary:

Good for: performance, reliability, andefficiency assessment


28/312

Flow Testing

Do one thing after another

- Define test procedures or high level cases that

incorporate many events and states.

- The events may be in series, parallel, or somecombination thereof.

- Dont reset the system between events.

Key Idea:

Summary:

Good for: finding problems fast(however, bug analysis is more difficult)

Scenario Testing

Test to a compelling story

- Design tests that involve meaningful and complexinteractions with the product.

- A good scenario test is a plausible story of howsomeone who matters might do something thatmatters with the product.

- Incorporate multiple elements of the product, and

data that is realistically complex.

Key Idea:

Summary:

Good for: Finding problems that seem important


29/312

User Testing

Involve the users

- Identify categories of users.

- Understand favored and disfavored users.

-Find these users and have them do testing orhelp you design tests.

-User testing is powerful when you involve avariety of users.

Key Idea:

Summary:

Good for: all purposes

Regression Testing

Test the changes

- Identify what product elements changed.

- Identify what elements could have been

impacted by the changes.

- Coverage strategy:

e.g., recent bug fixes, past bug fixes,

likely elements, all elements

Key Idea:

Summary:

Good for: managing risks related to productenhancement


30/313

Claims Testing

Verify every claim

- Identify specifications (implicit or explicit).

- Analyze individual claims about the product.

- Ask customer to clarify vague claims.

- Verify each claim.

- Expect the specification and product to be

brought into alignment.

Key Idea:

Summary:

Good for: simultaneously testing the product andspecification, while refining expectations

Risk Testing

Imagine a problem, then look for it.

- What kinds of problems could the product have?

- How would you detect them if they were there?

- Make a list of interesting problems and designtests specifically to reveal them.

- It may help to consult experts, designdocumentation, past bug reports, or apply risk

heuristics.

Key Idea:

Summary:

Good for: making best use of testing resources;leveraging experience


31/31

Software Testing 61

Random TestingRun a million different tests

- Look for an opportunity to automaticallygenerate thousands of slightly different tests.

- Create an automated, high speed evaluationstrategy.

- Write a program to generate, execute, andevaluate all the tests.

Key Idea:

Summary:

Good for: Assessing reliability across input and time.

02-Software Testing Introduction

Documents

Transcript of 02-Software Testing Introduction