Business Value ofAgile Testing

Scaling Up to Billion User Global Systems of Systems

Using TDD, CI, CD & DevOpsDr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFe

Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com

LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf

Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf

Author Background Gov’t contractor with 32+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe

2

Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 150 talks to 12,000+ people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at five Washington, DC-area universities

Today’s Global Marketplace Most of world’s population connected to Internet Systems must support billions of simultaneous users New approaches are needed to scale to global market

3Kemp, S. (2016). Digital in 2016: We are social's compendium of global digital, social, and mobile data, trends, and statistics. New York, NY: We Are Social, Inc.

NetworkComputer

Operating SystemMiddlewareApplications

APIsGUI

How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success

4Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.

Agile Traditional1 2 3 Faster

Early ROI

Lower Costs

Fewer Defects

Manageable Risk

Better Performance

Smaller Attack Surface

Late

No Value

Cost Overruns

Very Poor Quality

Uncontrollable Risk

Slowest Performance

More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction

MINIMIZES MAXIMIZES

JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes

Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success

5

Methods to “scope” project, product, or system “Key” is smallest possible scope with highest value Reduces cost, risk, time, failure, & tech. obsolescence

Barely Sufficient Design

INCREASES TESTABILITY, QUALITY, RELIABILITY, SECURITY, MORALE, MAINTAINABILITY, & SUCCESS

Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.

MINIMUM

MARKETABLE FEATURE- MMF -

AdvantageDifferenceRevenueProfitSavingsBrandLoyalty

MINIMUMVIABLE PRODUCT

- MVP -

GoalProcessFeaturesPrioritiesStory MapArchitecture

STORY MAPOR IMPACT MAP

- SM or IM -

GoalActors ImpactsDeliverablesMeasuresMilestones

VISIONSTATEMENT

- VS -

For <customer>Who <needs it>The <product> Is a <benefit>That <customer>Unlike <other>Ours <different>

MICRO-SERVICE- MS -

PurposeAutomatedUnique IndependentResilientEcosystemConsumer

6

Capability #1

● Feature 1● Feature 2● Feature 3● Feature 4● Feature 5● Feature 6● Feature 7

Capability #2

● Feature 8● Feature 9● Feature 10● Feature 11● Feature 12● Feature 13● Feature 14

Capability #3

● Feature 15● Feature 16● Feature 17● Feature 18● Feature 19● Feature 20● Feature 21

Capability #4

● Feature 22● Feature 23● Feature 24● Feature 25● Feature 26● Feature 27● Feature 28

Capability #5

● Feature 29● Feature 30● Feature 31● Feature 32● Feature 33● Feature 34● Feature 35

Capability #6

● Feature 36● Feature 37● Feature 38● Feature 39● Feature 40● Feature 41● Feature 42

Capability #7

● Feature 43● Feature 44● Feature 45● Feature 46● Feature 47● Feature 48● Feature 49

1

2 3

4

5 6

7

8 9

10

11 12

13

14 15

16

17 18

19

20 21

Evolving “Unified/Integrated” Enterprise Data Model

“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)

ETL

A A

B C

D E F

G H I J K

A

B C

D E F

A

B C

D E

A

B C

D

A

B C

A

B

“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server

“Leased” DWA/HPC/Cloud Services

Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7

ETL ETL ETL ETL ETL ETL

Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.

(for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total)

Organize needs into capabilities, features, and stories Prioritize features, group releases, and initiate sprints Develop minimum set of features with highest value

Agile Systems Development

Release

Release

Release

ReleaseMMF- or -MVP

What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components

7Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

AGILE TESTING- Early Incremental Testing -

TRADITIONAL TESTING- Late Big Bang Integration Testing -

Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at TimeCode is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports

Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One TimeCode Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports

Code Automatically DeployedLate Defects Freeze Projects

Thousands of TestsContinuously Executed

No More Late BigBang Integration

User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur

8Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.

BuildIntegration

Server

VersionControlServer

BuildScripts

UsesWatches

BuildStatus

ProvidesDeveloper A

Developer B

Developer C

CommitsChanges

CommitsChanges

CommitsChanges

Builds

Database

Analysis

Testing

Reporting

Documentation

Deployment

Early, Automated, Fast,Efficient, & Repeatable

Constant ReadinessState & CM Control

Lean, Waste Free, Low WIP,No Deadlocked Test Queues

Rapidly & SuccessfullyDev. Complex Systems

Agile Testing—Workflow

9

Traditional vs. Agile Cumulative Flow

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Traditional Cumulative Flow Agile Cumulative Flow

Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times

Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

Agile Testing—Workflow—Cont’d

BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated

10Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.

ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system

11Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

ALL DEVELOPERS RUN PRIVATE BUILDS

DEVELOPERS COMMIT CODE TO VERSION CONTROL

INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY

100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD

A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD

FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY

REPORTS AUTOMATICALLY GENERATED & REVIEWED

Agile CI consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.

12

Practice

Building

Database

Inspections

Testing

Feedback

Documentation

Deployment

Description

Frequently assembling products and services to ensure delivery readiness

Frequently generating/analyzing database schemas, queries, and forms

Frequently performing automated static analysis of product/service quality

Frequently performing automated dynamic product and service evaluation

Frequently generating automated status reports/messages for all stakeholders

Frequently performing automated technical/customer document generation

Frequently performing automated delivery of products/services to end users

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

PRACTICES—Continuous Integration

Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline

13Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.

CoQ

• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK

ENTERPRISE—Continuous Delivery

Agile CD consists of seven broad practices Automated acceptance, load, performance, etc. Include packaging, pre-production test, C&A, etc.

14

Practice

Packaging

Acceptance

Load Test

Performance

Pre-Production

Certification

Deployment

Description

Frequently generating system images for pre-production testing & checkout

Frequently performing automated system & user acceptance testing

Frequently performing automated system load, stress, & capacity testing

Frequently performing automated system user & technical performance testing

Frequently performing automated pre-production tests prior to final deployment

Frequently performing automated system certification & accreditation tests

Frequently generating product images for pre-deployment testing & checkout

Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.

PRACTICES—Continuous Delivery

Created by Patrick Debois of Jedi BVBA in 2007 Collaboration of developers & infrastructure people Goal to automate the deployment to end-user devices

15Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.

GLOBAL—Development Operations

Agile DevOps consists of seven broad practices Automated sys admin, CM, building, monitor, etc. Include virtualization, containerize, deployment, etc.

16

Practice

Sys Admin

Config. Mgt.

Host Builds

Virtualization

Containerize

Deployment

Monitor & Supp

Description

Frequently performing automated system administration tasks, i.e., scripting

Frequently performing automated infrastructure config. mgt./version control

Frequently performing automated system and server host builds and config.

Frequently performing automated system, server, & net virtualization services

Frequently performing automated software and Microservices containerization

Frequently generating final end-user system & software images for distribution

Frequently performing automated metrics collection & deployment monitoring

Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.

PRACTICES—Development Operations

Agile Testing Metrics—Example

17Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.

Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing

18

Agile Testing—CI Statistics

19

Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation

Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.

TYPE METRIC MANUAL DEVOPS MAJOR GAINS

CYCLE TIME

IMPROVEMENTS

Build Time 40 Hours 3 Hours 13 x

No. Builds 1-2 per Day 10-15 per Day 8 x

Feedback 1 per Day 100 per Day 100 x

Regression Testing 240 Hours 24 Hours 10 x

DEVELOPMENT

COST EFFORT

DISTRIBUTION

Integration 10% 2% 5 x

Planning 20% 5% 4 x

Porting 25% 15% 2 x

Support 25% 5% 5 x

Testing 15% 5% 3 x

Innovation 5% 40% 8 x

Agile Testing—CD Statistics

Assembla went from 2 to 45 releases every month 15K Google developers run 120 million tests per day 30K+ Amazon developers deliver 8,600 releases a day

20Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.

62x FasterU.S. DoD

IT Project

3,645x FasterU.S. DoD

IT Project

Agile Testing—DevOps Statistics

Activity Def CoQ DevOps Economics Hours ROIDevelopment Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%

Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%

Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%

Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%

"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%

Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%

Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a

21

Agile testing is orders-of-magnitude more efficient Based on millions of automated tests run in seconds One-touch auto-delivery to billions of global end-users

Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com

Agile Testing—DevOps CoQ

22

Agile Tools“Periodic Table of Testing Automation”

XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.

Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention

23Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

MICRO ADJUSTMENTS- Focused Impact Tuning-

MACRO ADJUSTMENTS- Wide Impact Tuning-

Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test LibrariesReduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware

In-Memory CompilationParallelize Test RunsPre-Install Test LibrariesRemove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process

Agile Testing—Scaling Practices

Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing

24

Technical BarriersOrganizational BarriersDevelopers don’t want to test

· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing

Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture

··

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Agile Testing—Common Barriers

Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project

25Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.

What’s the Bottom Line?“Agile Testing Done Early & Often”

Agile TestingTraditional TestingDramatically reduces risks

· Automates manual processes· Instant verification & validation· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability

Late defect discovery· Low quality software· Poor project visibility· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure

··

MIT’s Ten (10) LAWS OF SIMPLICITY One must think and act small to accomplish big things Slow down to speed up, speed up ‘til wheels come off Scaling up lowers productivity, quality, & business value

26Maeda, J. (2006). The laws of simplicity: Design, technology, business, and life. Cambridge, MA: MIT Press.

REDUCE - The simplest way to achieve simplicity is through thoughtful reduction.

ORGANIZE - Organization makes a system of many appear fewer.

TIME - Savings in time feel like simplicity.

LEARN - Knowledge makes everything simpler.

DIFFERENCES - Simplicity and complexity need each other.

CONTEXT - What lies in the periphery of simplicity is definitely not peripheral.

EMOTION - More emotions are better than less.

TRUST - In simplicity we trust.

FAILURE - Some things can never be made simple.

THE ONE - Simplicity is about subtracting the obvious and adding the meaningful.

Conclusion Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality

27Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdfRico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdfRico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf

What How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process

Customer Involve customers early and often throughout development Early feedback

Prioritize Identify highest-priority, value-adding business needs Focus resources

Descope Descope complex programs by an order of magnitude Simplify problem

Decompose Divide the remaining scope into smaller batches Manageable pieces

Iterate Implement pieces one at a time over long periods of time Diffuse risk

Leanness Architect and design the system one iteration at a time JIT waste-free design

Swarm Implement each component in small cross-functional teams Knowledge transfer

Collaborate Use frequent informal communications as often as possible Efficient data transfer

Test Early Incrementally test each component as it is developed Early verification

Test Often Perform system-level regression testing every few minutes Early validation

Adapt Frequently identify optimal process and product solutions Improve performance

Dave’s PROFESSIONAL CAPABILITIES

28

SoftwareQuality

Mgt.

TechnicalProject

Mgt.

SoftwareDevelopment

Methods

Strategy &Roadmapping

SystemsEngineering

CostEstimating

Acquisition &Contracting

OrganizationChange

Lean, Kanban,& Six Sigma

Modeling &Simulations

Big Data,Cloud, NoSQL

WorkflowAutomation

Metrics,Models, & SPC

BPR, IDEF0,& DoDAF

DoD 5000,TRA, & SRA

PSP, TSP, &Code Reviews

CMMI &ISO 9001

InnovationManagement

Statistics, CFA,EFA, & SEM

ResearchMethods

EvolutionaryDesign

Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.

Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.

STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.

● Data mining. Metrics, benchmarks, & performance.● Simplification. Refactoring, refinement, & streamlining.● Assessments. Audits, reviews, appraisals, & risk analysis.● Coaching. Diagnosing, debugging, & restarting stalled projects.● Business cases. Cost, benefit, & return-on-investment (ROI) analysis.● Communications. Executive summaries, white papers, & lightning talks.● Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.

PMP, CSEP,FCP, FCT

ACP, CSM,& SAFE

33 YEARSIN IT

INDUSTRY

Books on ROI of SW Methods Guides to software methods for business leaders Communicates the business value of IT approaches Rosetta stones to unlocking ROI of software methods

http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)

29