Evolution of the DevOps Quality Management Office

Evolution of the DevOps

Quality Management Office

May 2015

2 Copyright © 2015 Capgemini. All Rights Reserved

Evolution of the DevOps Quality Management Office | May 2015

How do we position testing ? DevOps Quality Management Office(QMO)

Vision: Continuous business driven testing to shrink development to operations time at continuous quality

DevOps driven Testing SLA’s

Shift left and right with Continuous Test

Four C’s – Continuous Integrate, deliver, deploy & monitor

Lean techniques & Agile methods

Standardization

Prepackaged workbenches

Hybrid test organization

QE,TDD,BDD

Test & Services virtualization

Environment provisioning & Test Data management

Test optimization

OATS & Risk based testing

Development Test Operations



What are our levers ?

Continuous quality with continuous delivery

>99% Application

availability

>30% Throughput

improvement

98% Prevention of

high severity defects

>30% Time-to-

Market

Success Levers

Advanced Intermediate Basic

70+%

50%

40+%

70+%

100%

100%

60%

60+%

50%

30%

40%

40%

70%

50%

30%

70%

Lifecycle Automation

Continuous Test

Lean and Agile adoption

Test virtualization

Test optimization

Standardization

Continuous monitoring

QE leverage

Functional, non functional, Data, UAT

Build, integrate and deliver

Kanban, Scrum, Safe, XP, SaFe

Services and Environment automation

Risk based testing and OA leverage

Model based work benches

Availability & sentiment analysis

TDD, BDD, Unit automation



What is the difference in traditional versus DevOps testing?

Process

Skills and People

Lean techniques and Agile

Continuous Test – Build, Integrate, Deliver, Deploy

Time to Market – Reduce time between development and operations

Standardization – Model-based testing

Optimization – Risk-based testing, OATS

Environment specialists

Services automation specialist

QE

Emerging Technology Adoption

Test virtualization – Services & Environment

Sentimental analysis and Continuous monitoring

Cloud adoption

DevOps Traditional Testing



What Speed is Right for an Test Organization?

It depends*…..

And it is a journey of continuous improvement…

*By Technology Fast Faster Fastest

Back-end apps & legacy

Front-end, client server, ERP

Web and mobile apps

Methodology

Waterfall and basic agile

Enterprise Agile

Test methods

Black box, Build and Integrate

Continuous test, white box, service

virtualization

Continuous deploy, environment set up

automated

Env virtualization, cloud technology

Release frequency



How do we decide? QMO advisory at a strategic level

DevOps Strategy assessment to determine objectives (Throughput, Availability, Time to market) Step

1

DevOps QMO assessment to identify bottlenecks in testing mapping to delivery bottlenecks Step

2

Leverage Kanban, VoC, Value Stream Mapping, DSM, Risk based technique Step

3

Identify no more than 5 test themes (Examples include test environment, automation, unit testing, service

virtualization)

Step

4

Build roadmaps accompanied by a business case (Roadmaps include organization, process and tools

transformation)

Step

5



With Agile test transformation as a strong lever

What is it? Benchmarks 17 key process areas in Agile testing.

How to use? Use to create agile test transformation roadmap. Process and automation tool kit reduces

implementation time to the tune of 30%

When to use? For quick assessments during transformation roadmap creation. Periodic assessments for

ongoing improvement in delivery.

Current State Future State

Aware of Following Excelling



And opportunities to Continuously test

Continuous

Integration Sprint

Product

Backlog

Priorities Spring

Backlog

Product

(Particularly usable)

Acceptance

Tests

Deployment

(production)

To Do… To Do…

Continuous

Delivery Acceptance

Tests

Deployment

(production) Sprint

Product

Backlog

Priorities Spring

Backlog

To Do…

Product


Continuous

Deployment

Acceptance

Tests

Deployment

(production) Sprint

Product

Backlog

Priorities Spring

Backlog Product




Building on our automation test types – Test Left and Right

Requirements Architecture & Design Development Testing Release

Sh

ift

Left

Test Driven Development

Automation: 80 – 100%

Unit Tests


Behavior driven development


Service Virtualization

Automation 40 – 60%

Non functional tests

Automation: 80-100%

Performance Tests

Automation; 100% Regression test


Test Management/Defect

Management


Sprint Acceptance Tests


Exploratory tests


Functional Tests


Sh

ift Rig

ht

UAT Automation

40 – 60%

Continuous Monitoring

Auto: 40 – 60%

Continuous Integration

Auto: 80 – 100%

Continuous Delivery

Auto:10-60%

Environment virtualization

Auto: 40 – 60%



With tools to continuously deliver

Assess Agile

QBP

Implement

Agile Toolkit

Manage QMO

Tools SLA’s

People

Perforce

TFS Code

SVN/GIT

Eclipse

ClearCase

Subversion

VStudio

IntelliJ

Docker

Nexus

Puppet

Chef

Third Party & Open

Source

Accurev

Mercurial

Hudsonl

Jenkins

TFS Build

Bamboo

El. Cloud

TestNG

NUnit

MsTest

Cobertura

JaCoCo

SAP SCM

Selenium

NCover

Ant Helion

OpenStack

Amazon

Ansible Maven

vCenter

JUnit

Netbeans Salt

..

Production Staging QA Dev

HP

Ag

ile M

ana

ge

r

Continuous Assessment

Continuous Delivery

Continuous Testing

Continuous Integration

HP Integrations to 3rd party & Open Source

HP ALM, Mobile Center, LR/Storm Runner, UFT, Fortify, NV, SV

HP CODAR

HP AppPulse, Real User Monitoring, SiteScope, Diagnostics

HP CSA

HP DevOps



Sample Use Case

HP Agile

Manager

HP Storm

Runner Load

HP Performance /HP

LoadRunner

HP

ALM

HP Unified Functional

testing

Continuous Testing, Delivery and Assessment

HP CODAR HP BSM HP AppPulse Mobile HP Cloud Service

Automation

HP AppPulse

Designer

Orchestrator

Provisioning engine

OOTB integrations

App Deployer

Configuration models

1 2 3



Measuring DevOps driven QMO SLA’s

DevOps SLA’s QMO SLA’s Testing KPI’s

Increase in number of

releases

Application availability

>99%

Reduce Mean time to

recover

Drive operational

efficiencies

Reduction in test cycle time

• Application Quality

• Speed and quality

Operational efficiencies

Reduce cycle time

Zero critical defects in

production

Phase containment –Shift

left

Reduce testing during

mean time to recover

Reduction in testing costs

release on release



With a clear vision of success

Test Execution Productivity

Velocity

Defect leakage

Severity 1 and 2 defects

Features delivered per release

Effort Variance

Schedule Variance

Number of Releases / Year

Reduction in Defects and Support Calls

Release Predictability Measure

Commitment Reliability

Sprint /

Iteration

Release

Engagement

Level

Metrics

Business

KPI

Sample Dashboards

Effort Manual VS Automation Number of releases YoY Defect Leakage to Sprint Number of support ticket YoY

-1064

462

2026.4

-2,000

-1,000

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

1 2 3 4 5

Effort manual

Effort Automation

Difference

0

2

4

6

8

10

12

14

1 2 3 4 5

No of releases

0

10

20

30

40

50

60

70

80

1 2 3 4 5

Defects

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5

No of Tickets



Putting it together – Sample roadmap

Deployments in Months Deployments in Weeks

Automated functional testing

Automated non-functional testing: performance, security

Automated test lifecycle management

Agile for programs

TDD and BDD

Service virtualization

Continuous delivery

Enterprise Agile

Overall: Much higher degree of automation

Continuous deployment

Full environment virtualization

Automation of UAT and exploratory testing

FA

ST

F

AS

TE

R

FA

ST

ES

T

The information contained in this presentation is proprietary and confidential.

Copyright © 2015 Capgemini. All rights reserved.

Rightshore® is a trademark belonging to Capgemini.

www.capgemini.com

About Capgemini

With more than 145,000 people in over 40 countries, Capgemini is one of the

world's foremost providers of consulting, technology and outsourcing services. The

Group reported 2014 global revenues of EUR 10.573 billion.

Together with its clients, Capgemini creates and delivers business and technology

solutions that fit their needs and drive the results they want. A deeply multicultural

organization, Capgemini has developed its own way of working, the Collaborative

Business Experience™, and draws on Rightshore®, its worldwide delivery model.

Learn more about us at www.capgemini.com.

http://www.capgemini.com/

http://www.facebook.com/Capgemini

http://www.linkedin.com/company/capgemini

http://www.twitter.com/capgemini

http://www.youtube.com/capgemini

http://www.slideshare.net/capgemini

http://www.capgemini.com/about/how-we-work/rightshorer





Evolution of the DevOps Quality Management Office

Technology

Transcript of Evolution of the DevOps Quality Management Office