Slide 1 Testing Workflow Purpose l Purpose Verify the result from implementation by testing each...

Testing Workflow Purpose

Purpose• Verify the result from implementation by testing

each build• Plan the tests in each iteration• Integration tests for every build within theiteration• System tests for the end of the iteration

• · Design and implement tests by creating• test cases that specify what to test• test procedures that specify how to perform the tests• executable test components to automate the tests

Testing Workflow Flow

Testing Workflow with Phases

Testing Workflow in Phases

Generally, when there is an implementationresult, there is a test --- so Test at each build

Inception phase: initial test planning, prototype testingElaboration phase: test architectural baselineConstruction phase: significant testing at each buildTransition phase: re-test fixes and regression tests

Testing Workflow Other Tasks

Regression tests: in a new build, re-apply tests from old builds to make sure nothing “broke” in new build

Evolve test modelcreate new test cases for next buildrefine prior test cases into regression testsremove obsolete tests and corresponding test procedures and components

Testing Workflow Models

Test model artifactsTest casesTest proceduresTest componentsTest subsystem packages for complex test

Other artifactsTest PlanDefectsTest Evaluation

Testing Workflow Model

TEST MODEL

Describes how executable components (e.g.,builds) in the implementation model are tested byintegration tests and system tests

Describes how specific system aspects are to betested

e.g., usability testing of user interface oruser manual

Testing Workflow – Test Case

TEST CASE

Specifies one way to test the systemWhat to test (requirements)With which inputsExpected resultsTest conditions and constraints

Testing Workflow Test Cases

TEST CASE EXAMPLES Test a use case or a scenario in a use case

(black box)§ actor interaction, pre-conditions,action sequences, post-conditions

Test a use-case realization (white box)

verify interaction between components. Installation tests, configuration tests, negative(abuse) tests, stress tests

Organize similar tests in input/result matrix Family of test for different paths through a use case

Testing Workflow Special TestsInstallation tests verify the system can be installed on the customer platform and that it executes correctly when installed

Configuration tests verify the system works correctly in different configurations

Negative tests (abuse tests) what causes failureTest outside the intended design, e.g., wrong

configuration, wrong input, etc.Reveals weaknesses in design

Stress tests show problems with insufficientor shared resources

Testing Workflow ProceduresSpecify how to perform one or several test cases orparts of them Instructions for tester in manual test case Instructions for interacting with test automation tool to create executable test components, then instructions to integrate and execute those test components

One test procedure may cover several test cases

One test case may require several test procedures Test procedure instructions often reflect flow-of events descriptions, including input values, how to enter input values, and how to verify results

Testing Workflow ComponentsAutomate one or several test procedures or parts

Provide test inputs, control and monitor testexecution, and possibly analyze and report results

Developed with scripting languages, programminglanguages, or test automation tools

test drivers, test harnesses, and test scripts

Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

Testing Workflow PlanDescribes testing strategies, resources and schedule· Strategy

What kind of tests to perform for each iterationTest objectivesRequired level of test and code coverageTest acceptance criteriaTrade-off cost vs. improved quality and reduced risk

Test Plan may be defined in conjunction with the model artifacts from other workflows

analysis (black box)design and implementation (white box)

“Tests” may include artifact reviews

Testing Workflow ResultsResults are the output data of the test and/or the behavior of the system during test

Analysis of results: pass or defectDefect: a system anomaly (software fault or adefect found in artifact review) capture symptom insufficient detail to allow problem resolutioncapture defect in problem and resolution tracking

system

Test evaluation: summary of overall test resultstest-case coverage, code coveragedefects and defect status

Testing Workflow Results

Testing Workflow Workers

Test designer: responsible for integrity of the test modelplan tests: test goals and scheduleselect and define test cases, test proceduresevaluate test results

Component engineer: responsible for development of test components

that automate some of thetest proceduresconsider independent test

Testing Workflow Workers

Integration tester: perform integration tests and report defectsmay be same person as system integrator inimplementation workflow

System tester: perform system tests and report defectsfocus on externally observable behaviorconsider users, beta customers, etc. as testers

Testing Workflow – THE PLAN

Plan the Tests Strategy and ResourcesDefine Integration TestsDefine System TestsDesign Regression TestsDefine and Mutate Test CasesImplement TestsEvaluate Tests

Testing Workflow – THE PLANPlan the Tests Strategy and Resources

Describe testing strategy Inputs: use case model, supplemental requirements,

design modelDevelop test cases and procedures that

have a minimum of overlaptest the most important test casestest high risk requirement items

Estimate resources Estimate human and system resources

Schedule the testing effort


Define Integration Tests

Derive test cases from use-case realizations – design (component interaction)

Develop tests with minimal overlap that test aninteresting path or scenario through use case

Primarily consider interaction diagramsCombinations of actor input, output and system

start state that lead to interesting scenarios that employ the classes that participate in the diagrams


Define System TestsTest system as a whole Test combinations of use cases instantiated under different conditions

Hardware configurationsSystem loadsNumber of actorsDatabase size, etc.

Prioritize combinations required or are likely to function in parallel likely to influence each other if in parallelinvolve multiple processes frequently use system resources or use them in complex and unpredictable ways


Design Regression Tests

Modify test cases from earlier builds to be usedas regression tests in subsequent builds

Make more flexible and resilient to changes to the system under test

Specifically address previously found defectsAutomate where possible


Define and Mutate Test Cases

Work test case by test case and suggest testingprocedures for each one

Reuse existing test casesMay need to modify them to be reusable

Align test procedures to subsystems under testMinimize overlapIsolates change to test resulting from changeto subsystem


Implement Tests

Create test components from test procedures With test automation tools

Perform or specify actions Record actions as a test component

When programming explicitlyDesign and implement using test procedures

Provide input and output data generation, visualization, interpretation and management utilities

Use spreadsheets and database applications


Evaluate TestsPerform the integration tests relevant to a build

Manually perform test proceduresCompare test results with the expected test resultsInvestigate test results that deviate from the expectedFaulty system and/or faulty test?

Report defects to the component engineer responsible forthe components likely to contain the fault

Report defects to the test designers for evaluation of overall test results


Evaluate Tests (cont)Test designers prepare metrics of test acceptance withrespect to system quality level

Test completeness and coveragePercentage of test cases executedPercentage of code and paths tested

ReliabilityCategorize defect severityAnalyze trends of discovered defects

Compare with prior experience to predict amount of testing still required to reach an acceptable defect leve


Evaluate Tests (cont)

Suggest actionsSuggest further testsRelax test criteriaDeliver parts of the system that passed; rework and retest parts that failed

Document test completeness, reliability and actions

Testing Workflow – SUMMARY

Test Model Test cases specify what to test Test procedures specify how to perform tests Test components automate test procedures

Test Plan

Test EvaluationDefects to be fed into other core work

Slide 1 Testing Workflow Purpose l Purpose Verify the result from implementation by testing each...

Documents

Transcript of Slide 1 Testing Workflow Purpose l Purpose Verify the result from implementation by testing each...