Testing Workflow

34
Testing Workflow In the Unified Process and Agile/Scrum processes

description

Testing Workflow. In the Unified Process and Agile/Scrum processes. Testing Workflow: Purpose. Verify the result from implementation by testing each build Plan the tests in each iteration Integration tests for every build within the iteration System tests for the end of the iteration - PowerPoint PPT Presentation

Transcript of Testing Workflow

Page 1: Testing Workflow

Testing Workflow

In the Unified Process and Agile/Scrum processes

Page 2: Testing Workflow

Testing Workflow: PurposePurpose

• Verify the result from implementation by testing each build– Plan the tests in each iteration– Integration tests for every build within the

iteration– System tests for the end of the iteration

• Design and implement tests by creating– test cases that specify what to test– test procedures that specify how to perform the

tests– executable test components to automate the

tests

Page 3: Testing Workflow

Review: Kinds of testing

• Unit testing– Does an individual software component

(unit) live up to its contract?– Typically done by the same person who

codes the given unit– Tests become part of regression test suite

• Integration testing• System testing• Regression testing

Page 4: Testing Workflow

Where do unit tests fit into UP?

• Unit tests are part of the implementation workflow

• They come before the integration testing

• They are typically designed and written by the programmer who writes the given module/unit

• (See separate slides on unit testing.)

Page 5: Testing Workflow

Review: Kinds of testing

• Unit testing• Integration testing– Does a collection of modules function

correctly together as a group?

• System testing• Regression testing

Page 6: Testing Workflow

Review: Kinds of testing

• Unit testing• Integration testing• System testing– Does the entire system meet the end

user’s requirements?

• Regression testing

Page 7: Testing Workflow

Review: Kinds of testing

• Unit testing– Shows that individual software

components (units) live up to their contracts

– Typically done by the same person who codes the given unit

– Tests become part of regression test suite

• Integration testing• System testing• Regression testing

Page 8: Testing Workflow

Testing Workflow: FlowFlow

Page 9: Testing Workflow

Testing Workflow in UP PhasesUP Phases

Page 10: Testing Workflow

Testing Workflow in PhasesGenerally, when there is an implementationresult, there is a test --- so Test at each build

Inception phase: initial test planning, prototype testingElaboration phase: test architectural baselineConstruction phase: significant testing at each buildTransition phase: re-test fixes and regression tests

Page 11: Testing Workflow

Testing Workflow: Other Tasks

Regression tests:in a new build, re-apply tests from old builds to make sure nothing “broke” in new build

Evolve test modelcreate new test cases for next buildrefine prior test cases into regression testsremove obsolete tests and correspondingtest procedures and components

Page 12: Testing Workflow

Testing Workflow: ModelsModels

Test model artifactsTest casesTest proceduresTest componentsTest subsystem packages for complex test

Other artifactsTest PlanList of product DefectsTest Evaluation

Page 13: Testing Workflow

Testing Workflow: ModelModel

Describes how executable components (e.g.,builds) in the implementation model are tested byintegration tests and system tests

Describes how specific system aspects are to betested

e.g., usability testing of user interface oruser manual

Page 14: Testing Workflow

Testing Workflow: Test CaseTest Case

Specifies one way to test the systemWhat to test (requirements)With which inputsExpected resultsTest conditions and constraints

Page 15: Testing Workflow

Test Case ExamplesTest a use case or a scenario in a use case

(black box)actor interaction, pre-conditions,action sequences, post-conditions

Test a use-case realization (white box)

verify interaction between components installation tests, configuration tests, negative(abuse) tests, stress tests

Organize similar tests in input/result matrixFamily of tests for different paths through a use case

Page 16: Testing Workflow

Special Tests

Verify the system can be installed on the customer platform and that it executes correctly when installed

Configuration tests verify the system works correctly in different configurations

Negative tests (abuse tests) discover what causes failureTest outside the intended design, e.g., wrong

configuration, wrong input, etc.Reveals weaknesses in design

Stress tests show problems with insufficientor shared resources

Page 17: Testing Workflow

Testing Workflow: Test ProceduresTest Procedures

Specify how to perform one or several test cases orparts of them•Instructions for tester in manual test case•Instructions for interacting with test automation tool tocreate executable test components, then instructions to integrate and execute those test components•One test procedure may cover several test cases•One test case may require several test procedures•Test procedure instructions often reflect flow-of events descriptions, including input values, how to enter input values, and how to verify results

Page 18: Testing Workflow

Testing Workflow: Test Test ComponentsComponents

Automate one or several test procedures or partsProvide test inputs, control and monitor testexecution, and possibly analyze and report results

Developed with scripting languages, programminglanguages, or test automation tools

test drivers, test harnesses, and test scripts

Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

Page 19: Testing Workflow

Testing Workflow: Test PlanTest PlanDescribes testing strategies, resources and scheduleStrategy

What kind of tests to perform for each iterationTest objectivesRequired level of test and code coverageTest acceptance criteriaTrade-off cost vs. improved quality and reduced

risk

Test Plan may be defined in conjunction with the model Artifacts from other workflows

analysis (black box)design and implementation (white box)

“Tests” may include artifact reviews

Page 20: Testing Workflow

Testing Workflow: Test ResultsTest Results

Results are the output data of the test and/or the behavior of the system during test

Analysis of results: pass or defectDefect: a system anomaly (software fault or adefect found in artifact review) • Capture symptom in sufficient detail to allow problem resolution• Record defect in problem and resolution tracking

system

Test evaluation: summary of overall test resultstest-case coverage, code coveragedefects and defect status

Page 21: Testing Workflow

Testing Workflow: Test Results

Page 22: Testing Workflow

Testing Workflow: Workers

Test designer: responsible for integrity of the test modelplan tests: test goals and scheduleselect and define test cases, test proceduresevaluate test results

Component engineer: responsible for development of test

components that automate some of thetest proceduresconsider independent test

Page 23: Testing Workflow

Testing Workflow: Workers

Integration tester:perform integration tests and report defectsmay be same person as system integrator inimplementation workflow

System tester: perform system tests and report defectsfocus on externally observable behaviorconsider users, beta customers, etc. as testers

Page 24: Testing Workflow

Testing Workflow – THE PLAN

Plan the Tests Strategy and ResourcesDefine Integration TestsDefine System TestsDesign Regression TestsDefine and Mutate Test CasesImplement TestsEvaluate Tests

Page 25: Testing Workflow

Testing Workflow – THE PLAN

Plan the Tests Strategy and Resources

Describe testing strategy Inputs: use case model, supplemental

requirements, design modelDevelop test cases and procedures that

have a minimum of overlaptest the most important test casestest high risk requirement items

Estimate resources Estimate human and system resources

Schedule the testing effort

Page 26: Testing Workflow

Testing Workflow – THE PLAN

Define Integration Tests

Derive test cases from use-case realizations – design (component interaction)

Develop tests with minimal overlap that test aninteresting path or scenario through use case

Primarily consider interaction diagramsCombinations of actor input, output and system

start state that lead to interesting scenarios that

employ the classes that participate in the diagrams

Page 27: Testing Workflow

Testing Workflow – THE PLAN

Define System TestsTest system as a whole Test combinations of use cases instantiated under different conditions

Hardware configurationsSystem loadsNumber of actorsDatabase size, etc.

Prioritize combinations required or are likely to function in parallel likely to influence each other if in parallelinvolve multiple processes frequently use system resources or use them in complex and unpredictable ways

Page 28: Testing Workflow

Testing Workflow – THE PLAN

Design Regression Tests

Modify test cases from earlier builds to be usedas regression tests in subsequent builds

Make more flexible and resilient to changes to the system under test

Specifically address previously found defectsAutomate where possible

Page 29: Testing Workflow

Testing Workflow – THE PLAN

Define and Mutate Test Cases

Work test case by test case and suggest testingprocedures for each one

Reuse existing test casesMay need to modify them to be reusable

Align test procedures to subsystems under testMinimize overlapIsolates change to test resulting from changeto subsystem

Page 30: Testing Workflow

Testing Workflow – THE PLAN

Implement Tests

Create test components from test procedures With test automation tools

Perform or specify actions Record actions as a test component

When programming explicitlyDesign and implement using test

procedures Provide input and output data generation, visualization, interpretation and management utilities

Use spreadsheets and database applications

Page 31: Testing Workflow

Testing Workflow – THE PLAN

Evaluate TestsPerform the integration tests relevant to a build

Manually perform test proceduresCompare test results with the expected test resultsInvestigate test results that deviate from the

expectedFaulty system and/or faulty test?

Report defects to the component engineer responsible forthe components likely to contain the fault

Report defects to the test designers for evaluation of overall test results

Page 32: Testing Workflow

Testing Workflow – THE PLAN

Evaluate Tests (cont)Test designers prepare metrics of test acceptance withrespect to system quality level

Test completeness and coveragePercentage of test cases executedPercentage of code and paths tested

ReliabilityCategorize defect severityAnalyze trends of discovered defects

Compare with prior experience to predict amount of testing still required to reach an acceptable defect level

Page 33: Testing Workflow

Testing Workflow – THE PLAN

Evaluate Tests (cont)

Suggest actionsSuggest further testsRelax test criteriaDeliver parts of the system that passed; rework and retest parts that failed

Document test completeness, reliability and actions

Page 34: Testing Workflow

Testing Workflow – SUMMARY

Test Model Test cases specify what to test Test procedures specify how to perform tests Test components automate test procedures

Test Plan

Test EvaluationDefects to be fed into other core work