Agile Kaizen - Continuous Improvement Far Beyond Retrospectives
Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE...
Transcript of Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE...
![Page 1: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/1.jpg)
Continuous Change-Driven Build Verification (ChDT™) Marc Hornbeek Manager, SCM Systems Spirent Communications
June 2011
![Page 2: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/2.jpg)
INTRODUCTION
![Page 3: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/3.jpg)
INTRODUCTION
Manager, SCM Systems, Spirent Communications • Network Test and Measurement Systems • Responsible for SCM Systems including automated BV • Practical “How-to” guides for Engineering management professionals
Perforce Environment
• 1.6TB, 36 MLOC, 9 apps, 15 platforms, 18 releases/year, 100 builds /day
• 3000 tests/build: Build Test Cycle: 23 hours -> 4 hours
![Page 4: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/4.jpg)
4
ChDT™ GAME-CHANGER FOR LARGE SCALE AGILE
• Agile Manifesto: – Early and continuous delivery of
valuable software – Deliver working software frequently – Working software is the primary
measure of progress.
• Functional testing is the standard of proof.
• Testing for large scale systems is a
bottleneck
• By dynamically testing code changes ChDT™ resolves this bottleneck.
![Page 5: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/5.jpg)
ChDT™ Methodology
![Page 6: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/6.jpg)
6
TYPICAL CODE / TEST FLOW
Automated Test Harness
Test Report
Software Build
PTest 01
PTest 02
PTest 03
PTest 04
PTest 05
95% 4% 1%
Test Case Database
Code
Pre-defined Test Suite
Source Code
Change Control
SCM
..yes but… L § time to run all the tests ? § equipment to run enough tests in parallel ? § time and effort to analyze all the results ?
Automation allows the tests to be run automatically …right?...
![Page 7: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/7.jpg)
7
ChDT™ CODE / TEST FLOW
Automated Test Harness
Test Case Database
Change-Driven Test (ChDT) Module
Targeted Test Report
PTest 01
PTest 02
PTest 03
PTest 04
PTest 05
95% 4% 1%
Generated Test Suite
Software Build
Source Code
Change Control
SCM
Code
CODTEF MATRIX
Test Cases
Cod
e M
odul
es
0.7
1.0
0.0
0.8
0.2
0.5
The Change-Driven Test ChDT™ Methodology
introduces a CODTEF™ Matrix into the code/test
flow. The matrix is used by the ChDT™ system to
automatically drive both test selection and test result
report generation according to continuously adjusting code change-to-test-to-
results correlation factors in the CODTEF™ Matrix.
ChDT and CODTEF are trademarks of Managecycle. (c) 2011 All rights reserved.
![Page 8: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/8.jpg)
8
TEST SELECTION METHODS COMPARED
“Empirical Study of Regression Test Selection Techniques – ACM Transactions” 1. Retest all - test everything regardless of changes 2. Hunches – human intuition 3. Randomly select tests 4. Minimalist - minimal code coverage 5. Dataflow technique - data interactions 6. Safe Techniques - cover each code segment deleted and added.
Change-driven: select tests based on specific code change information correlated to prior failure history of tests correlated to the changed modules
Efficient ? Yes No Risk? Low High Complexity? Low High Platform ? Independent Dependent
![Page 9: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/9.jpg)
9
DYNAMIC ADJUSTMENT
• Code-to-test CODTEF™ correlation factors automatically adjust – Past test results automatically affect future test selections – Automatic adjustments of test timings – Automatic adjustment of test reports – Automatic adjustments of code and test performance data
Build/Test Process
Input
Output
Automatic Adjustment
![Page 10: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/10.jpg)
IMPLEMENTATION FRAMEWORK
![Page 11: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/11.jpg)
11
PRE_REQUISITES FOR ChDT IMPLEMENTATION
Technology: • Source code management system • Automated test framework
Process: • Standalone automated test cases**
People: • Tools developers (SCM Framework
tools, DB design)
** Practical reference ** TSCRIPT™: A Practical System for Realizing
Exemplary Automated Test Scripts
![Page 12: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/12.jpg)
12
ChDT DATA FRAMEWORK
Framing the system around the product itself provides a logical framework.
There are choices: 1) Complete process-able definition of the product 2) Use attribute tags to define important (to test) characteristics 3) Tests themselves define the product behavior 4) Groups of modules and tests define major blocks of product behavior 5) Code itself describes the product behavior
ChDT is a system of many variables • Code variables • Test variables • Results variables • System variables • Computed variables
![Page 13: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/13.jpg)
13
DEVELOPMENT PHASES
Break up test suite into groups. Identify code group – test group relationships.
Tools: Automatically
identify group code changes, Launch specific test groups, Report results according to group owners
Identify code subgroups and initial test correlations. Identify and design specific tools changes.
Tools: Implement remaining
tools listed on prior chart except the advanced metrics
Analysis and design: Determine which advanced metrics will be used by the organization.
Tools: Create metric tools
using the data collected during the Change-Driven loops.
Phase 1 Code/Test
Group
Phase 2 Refined
Sub-Groups
Phase 3 Advanced
Metrics
![Page 14: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/14.jpg)
TOOLS
![Page 15: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/15.jpg)
15
ChDT™ ARCHITECTURE
Code Change Analyzer: • Code Change DB • Code deltas Test Selector: • Test Case Results DB • CODTEF™ Matrix • Test Case Selector System Infrastructure: • User inputs • Resource DB • Adjust CODTEF™
values Reports and Metrics: • Report Generators • Metrics calculators ** Practical reference **
TCHANGE™ : A Practical System For Implementing Change-Driven Test
Methodology
CODTEF MATRIXProcess
Test Cases
Cod
e M
odul
es
SCM
0.7
1.0
0.0
0.8
0.2
0.5
User Input Command Processor
Test Case
Selector
Test Report Generater
Test Harness
Test Report
ChDT Code Data
ChDT Testcase
Data
ChDTTest
ResultAnalyzer
Test CasesC
ode
Mod
ules
Code Change Analyzer
0%Code Module 01
2%Code Module 02
3%Code Module 03
0.1%Code Module 04
0%Code Module 05
Test Suite Generater
Test Case DB
ChDTModule
![Page 16: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/16.jpg)
16
CODTEF™ MATRIX
CODTEF™: Code/Test correlation Factor for each Code/Test combination
CODTEF MATRIX
Test Cases
Code
Mod
ules 0.7
1.0
0.0
0.8
0.2
0.5
CODTEF Value
Meaning
Blank unknown / not set 0.0 no correlation 0.5 some correlation 1.0 Direct correlation
INITIALIZING CODTEF™ 1) Manual Method: Coders/Testers fill in
the CODTEF values in the matrix – Groups of code/tests – Individual tests
2) Automatic – Set all CODTEF factors to the
average value of all CODTEF factors. ** (or 0.5 very initially until an average has been established)
3) Semi-automatic – Set some CODTEF factors using
method 1, then let the system complete the rest using method 2
![Page 17: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/17.jpg)
17
3. PASS:
New CODTEF = Prior CODTEF - δP(1- CODTEF), Minimum=0
where δP is the change rate of CODTEF for PASS that will match the change rate of FAIL if the system performs at the average level over time. δP= δF(δF/(1- δF)) (E.g. 0.03x(0.03/(1-0.03)) = 0.001 )
2. FAIL:
New CODTEF = Prior CODTEF + δF(1- CODTEF),
Maximum=1 where δF = average test failure rate (E.g. 0.03)
CODTEF™ ALGORITHM
1. Inconclusive: Do not adjust CODTEF Adjust CODTEF after Pass/Fail/Inconclusive verdict is established for each test:
0.0 1.0Δ δF
0.0 1.0ΔδP
CODTEF
CODTEF
![Page 18: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/18.jpg)
18
SYSTEM DATA
R : Resources available for test T : Time available for test Q : Quality level = minimum CODETEF™ value M : Must-do tests TE : Test Exclusion guard band
• % of tests allowed to be excluded from each level if T does not allow selection of all of them.
![Page 19: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/19.jpg)
19
TEST SELECTION ALGORITHM
Test Selection = TS1 +TS2 +TS3 +TS4 = = Test Selection Level 1 = Must-do tests
• Regardless of CODTEF™
+ Test Selection Level 2 = Recently Failed tests • Regardless of CODTEF™
+ Test Selection Level 3 = Best CODTEF™ tests • CODTEF™ > Q
+ Test Selection Level 4 = Additional tests if time allows • CODTEF™ < Q
![Page 20: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/20.jpg)
RESULTS AND METRICS
![Page 21: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/21.jpg)
21
METRICS DERIVED FROM CODTEF™ VALUES
Testers – High yield tests to be re-factored (high average CODTEF™) – Low yield tests to be pruned (low average CODTEF™)
Code Owners
– CODTEF™ is Q or higher • Eliminates redundant reports of test results to irrelevant owners
– Code reliability: re-factor error prone code modules (modules with high average CODTEF™ values)
QA Managers
– Increasing average CODTEF™ values indicate quality erosion
![Page 22: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/22.jpg)
22
RETURN ON INVESTMENT (ROI)
** Practical reference ** TCASE™: A Practical System for Creating $uccessful
Test Automation Business Cases
250 Developers3,150,000 LOC
4,550 test cases36 hours to run all tests4 hours average cycle test with ChDT
3,800 bugs fixed per year4 hours to fix bug found during ChDT cycle
20 hours to fix bug found after ChDT cycle30% fewer bugs escape integration after ChDT implements
$130,000 loaded labor rate $/year =$62.50 $/hour
$14,250,000 Cost of bugs before ChDT (3 years)$10,830,000 Cost of bugs after ChDT (3 years)$3,420,000 Cost savings of CHDT derived from lower cost of bugs (3 years)
$120,000 ChDT specific Tools labor (one time)$21,000 Reconfigure existing tests (one time)$7,500 ChDT sever (one time capital expense
$21,000 Annual maintenance$211,500 3 year cost
16 to 1 return$3,208,500 net cash equivalent returnedR
OI
COST
BENEFIT
![Page 23: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/23.jpg)
EXAMPLE SYSTEM
![Page 24: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/24.jpg)
SPIRENT BV SYSTEM
EXAMPLE SSCC RESULT FILE
<?xml version="1.0" ?> - <Report branch="mainline" startbuild="3.50.4330" endbuild="3.50.4810"> - <FileGroup name="IPS" owner="Marc Hornbeek"> <FilePath name="PortConfigs" changes="0" /> <FilePath name="Rfc2544Back2Back" changes="0" /> <FilePath name="Rfc2544Common" changes="0" /> <FilePath name="Rfc2544FrameLoss" changes="0" /> <FilePath name="Rfc2544Latency" changes="0" /> <FilePath name="Rfc2544Throughput" changes="0" />
</FileGroup>
- <FileGroup name="Routing" owner="Owner Name"> <FilePath name="bfd" changes="0" /> <FilePath name="bfd" changes="8" /> <FilePath name="eoam" changes="0" />
</FileGroup> </Report>
SSCC = Smartest Source Code Counting tool
![Page 25: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/25.jpg)
SUMMARY
![Page 26: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/26.jpg)
26
SUMMARY
Continuous Change-Driven Build Verification (ChDT™) methodology using the CODTEF™ matrix is a practical, platform independent, efficient and scalable solution to the build / verification loop dilemma.
![Page 27: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/27.jpg)
27
RELATED READING
• TCHANGE™ : A Practical System For Implementing Change-Driven Test Methodology, Marc Hornbeek, 2010
• A Systematic Review on regression test selection techniques; Emelie Engström, Per Runeson, Mats Skoglund, Department of Computer Science, Lund University, SE-221 00 Lund, Sweden
• An Optimized Change-Driven Regression Testing Selection Strategy for Binary JAVA Applications; Sheng Huang, Yang Chen, Jun Zhu, Zhong Jie Li, Hua Fang Tan; IBM China Research Laboratories, and Department of Computer Science, Tsinghua University Beijing
• An Empirical Study of Regression Test Selection Techniques; Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, Gregg Rothermel; Ohio State University, Oregon State University, and University of Maryland
• Market Survey of Device Software Testing Trends and Quality Concerns in the Embedded Industry, Wind River, June 2010
![Page 28: Continuous Change-Driven Build Verification (ChDT™)...4 ChDT GAME-CHANGER FOR LARGE SCALE AGILE • Agile Manifesto: – Early and continuous delivery of valuable software – Deliver](https://reader034.fdocuments.in/reader034/viewer/2022050110/5f47a5df1d01695d262249ff/html5/thumbnails/28.jpg)
THANK-YOU !
Thank-you for attending this session.
Marc Hornbeek
ü Please fill out an evaluation form. THANK-YOU !!
The End
Follow-up comments, questions and suggestions? [email protected]