How to Solve Problems of Testing in Agile Environments

World®’16

HowtoSolveProblemsofTestinginAgileEnvironmentsJonathonWright– DirectorofDigitalAssurance– CATechnologiesHuwPrice– VPofApplicationDelivery– CATechnologies

DO5X29E

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ForInformationalPurposesOnlyTermsofthisPresentation

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.


Abstract

Companiesadoptinganagileframeworkforsoftwaredevelopmentknowthatoneareastillcontributingtoslowdownsistesting.Inallitsforms—regression,system,performance,etc.—testingisnotoptimizedinthesoftwaredevelopmentlifecycle.Theimportanceoftestdatamanagement—havingtherightdatadeliveredtotherightplace,attherighttime—fortestingpurposesisoftenoverlooked.

Poortestcasedesignandtheinefficientprovisioningofpoorqualitydatameanthattestteamsfindthemselvesunabletokeepupwiththerateofchangewhilefullytestingsystems.Qualityisthencompromisedinfavourofdeliveringsoftwareontimeandwithinbudget.Organizationsthatwanttoimplementcontinuousdeliverymustrethinktheirtestinganddevelopmentprocessesratherthansimplyreshufflethem,andreconsidertheirapproachtotestdatamanagementandtestcasedesign.Byusinganend-to-endapproachtotesting,drivenbyrequirements,organizationscandoshift-lefttesting,mitigateriskandminimizedefectcreation,therebydeliveringqualitysoftwarefaster,andforlesscost.

HuwPriceCATechnologiesVicePresidentofApplicationDelivery

CATechnologiesDirectorofDigitalAssurance

JonathonWright


Agenda

INTRODUCTION

DIGITALEVOLUTION

COGNITIVEADAPTIVEADOPTION

SOLUTIONSTHINKING

DIGITALASSURANCE

CONTINUOUSADAPTIVETESTING

1

2

3

4

5

6


ShortenedReleaseCycles

DemandforHighQuality

BalancingToday’sDemands

Customerneeds MovingtowardsAgiledevelopment

Competitivepressures

Digitaltransformationinitiatives

Mosttestingismanual,notautomated

Bettertestdataneeded

ImpactsCustomerExperience


Businessesnolongerhavetheluxuryoftime.Risingcustomerexpectations,competitivethreats,andincreasedconsumerchoicehaveturnedrapiddeliveryanditerationofsoftwareapplicationsintoacompetitivedifferentiator.

Optimizingthedeliveryvaluechainhasbecomethenewmeanstobetterbusinessoutcomes.

TechRadar™:ContinuousSoftwareDelivery,Q22015,ForresterResearch,May1,2015

“

”

2014VansonBournestudycommissionedbyCA

UNPRECEDENTED

94%ofexecutivesfaceincreasedpressuretoreleaseappsmorequickly

“

TheJourneytoDigitalTransformation


CognitiveAdoption– ‘Evolution,OverRevolution’

1– “TheBattleforCompetitiveAdvantageintheAppEconomy”,OxfordEconomics,2015

VALU

E-DR

IVEN

TIME

1960s 1990s TODAY

ofenterprisesbelievethattheyneedvalue-drivenalgorithmicbusinessmodels torespondtodisruptioninthenext3years.

78%

LegacyisyourLegacy(API)

DigitalizationofyourCore(SOA)

Digital(CognitiveAdoption)


78% of Executives in most B2C industries say their business will be disrupted by digital technologies in the next 12 months. 20% say their business will be massively disrupted.

- Forrester State of Digital Business - Big Bang Disruptions - Larry Downes - RallyON Keynote


CompaniesLooktoDevOpsfortheAnswer

VansonBourne– DevOpssurvey2014

BenefitsofDevOpsPercentimprovementsinbusinessareas

Collabo

ratio

n

Appqu

ality

Custom

ers

New

services

Time-to-m

arket

Revenu

e

Costsa

vings

Deployfreq

uency

PlantoadoptDevOpsNext3to5years

15%88PERCENT

18% 18% 19% 19% 19%21% 21%


OurFocus…Dramatically ImproveTestinginDigital

Better. Value. Smarter.

*NationalInstituteofStandards&Technology

$60B $2.5BPerYearofITBudgetforQAandTesting*

PerYearonTestingTools

HowAboutFocusingonMinimumViableQuality?


Limitedtestresourcesand

lackofAutomation

ofalltestingisstillmanual70%

Source:BloorResearch,2014

TestDataforsetup&staging

oftimespentlookingfor50%

Source:BloorResearch,2014

AgileRequirementsDesigner,TDMandRA

Buildofvariousdev&testing

Environments

ofcriticaldependenciesareunavailable whendevandtestteamsneedthem

56%

Source:vokeMarketSnapshotTM Report:ServiceVirtualization– January2015

IsTestingstilltheBottleneck?

DigitalTesting


DESIGNOPS

ContinuousDelivery- Sub-ValueStreams

1ContinuousDelivery

2ContinuousTesting

3ContinuousStaging

4ContinuousRelease


idea

DEVELOPMENT OPERATIONS

‘Shift-Right’DigitalEcology

‘Shift-Up’DigitalAssurance

‘Shift-Left’DigitalEvolution

CUSTOMEREXPERIENCETESTING

DesignDevTestOpsAgilityacrossall4increasesspeed,reliabilityandefficiency

DESIGN

‘Shift-Down’DigitalArchaeology

ContinuousAdoptionCognitiveAdaptiveAdoption

ContinuousTesting

ContinuousImprovement

ContinuousDelivery

ContinuousIntelligence

ContinuousLearning

ContinuousInnovation

CognitiveAdaptiveTesting

CognitiveAdaptive

Improvement

CognitiveAdaptiveDelivery

CognitiveAdaptive

Intelligence

CognitiveAdaptiveLearning

CognitiveAdaptiveInnovation


WATERFALL/V-MODELDAD,LeSS&SAFE(v4)LEANSTARTUP/UX

Incubate Scale

FASTFEEDBACK

BUSINESSINTELLIGENCEREALTIMEPREDICTIVEPERSCRIPTIVE

Enhance

SLOWFEEDBACKWHATISHAPPENINGNOW?

WHATSHOULDWEDO?EXPERIMENT ASSURE


DigitalEvolutionfeat.DesignOps‘ShiftLeft’BecominganDigitallyDisruptiveBusiness

PORTFOLIOMANAGEMENT

Easilyandsecurelymanagefromsinglelocation&decreasecreationtime

by96%

LIFECYCLEVIRTUALIZATION

Simultaneouslyaccessandvirtualservicesacceleratingapp

developmentby97%

SCALEDAGILEMANAGEMENT

30–50%improvementsinbothcostanddeliverydurationcomparedtotraditionalprojects

SHIFTLEFT

Innovation….Faster


RapidEvolutionSolution&DesignThinking,LeanDigital,Customer&UserExperience(DX,CX&UX)

*DaveLandis,LitheSpeed.com,LeanUXprinciplesandDesignThinking

©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD


Build

Deliver

MonitorMeasure

Learn

Design

Make

Check

Think

Proactiveover

Reactive

Governanceover

Ownership

Maturityover

Mastery

Predictiveover

Continuous

Evolutionover

Revolution

EngineeringoverAgility

IntelligenceoverThings


INNOVATION(INNOVATION,GROWTH&MATURITY)FAST

INNOVATIONSLOW

INNOVATION

DECOMPOSITIONLIBERATIONIMMUNIZATION

Incubate Scale Enhance

WHATISHAPPENINGNOW?

EcologistDevScrum

Master

TherapistResearcher

Designer

DevOpsEngineer

DisciplinedAgilist

MakerCreative

InfrastructureEngineer

SDET

BA

Tester

ProductOwner PM

Ops

Support

FASTFEEDBACK

SLOWFEEDBACK

WHATHAPPENED?WHAT’SHAPPENINGNOW?WHATWILLHAPPEN?WHATCOULDHAPPEN?


UseCase- AgileArchitectDesigner

Wireframe(DEMO)

DigitalEvolution



IdeasBeliefs,Values,Culture

Heuristics

Behaviors

Value

AgileLean

Copyright©2016nuCognitiveLLC.AllRightsReserved.

MethodsandSkills

SolutionThinkingExploringSolutionsThinking(Business,Technology&Usage)


SolutionThinking

RayArell

http://www.solution-thinking.com



LIFECYCLEVIRTUALIZATION

Simulatemissingsystems,tosaveupto640Hrs.

DIGITALENGINEERING&AUTOMATION

Testless,covermore,automate100%oftests.

DATAVISUALIZATION&VIRTUALIZATION

Reducetimespentwaitingondataby95%.

SHIFTUP

VALUE-DRIVENMulti-ModalDelivery

(IncubatetoLead,ScaleforGrowthandEnhancetheCore)

DigitalAssurance‘Shift-Up’BecominganValue-DrivenBusiness


COGN

ITIVEAD

APTIVE

ADO

PTION

ManualContinuous

Autonomics

LeanEngineering

ConnectedIntelligence

BUSINESSVALUE

LEVEL1 LEVEL2 LEVEL3 LEVEL4 LEVEL5

?

OptimizedOrganizationforDesignOps

ContinuousInsight&Learning

ValueStreamMapping

Consistency&Collaboration

Heroes&HeavyLifting

DIGITA

LMAT

URITY

§ Insight-Driven(Predictive/Prescriptive)§ NoOpsorganized(IoT-Ops/CX-Ops/Sec-Ops)§ Self-healingend-to-endautonomicorchestration§ Microcontainerization&Microservicesenabled§ OpenInnovation&ConnectedIntelligence§ QuantumTeleportation(ShiftX)

Whereareyou?

MinimumViableQuality

§ Silos,manualhandovers,waterfallprocess§ Onerelease/year§ Monolithicapps§ Longtermproject/resourceplanning§ Errorpronedev/test/releaseprocesses

MinimumViableProduct

MinimumViableDelivery

OrganizationChange

MethodsandSkills Insight

Beliefs,Values,Culture

Value Behaviors Heuristics


ConnectedIntelligence

ReleaseManagement

EnvironmentManagement

ApplicationManagement

NetworkManagement

InfrastructureManagement

RepositoryManagement

DataManagement

Test Data LearnProbeCAAppExperienceAnalytics

ConnectedIntelligenceCognitiveAdaptiveInsight(CAI)


REALTIME,BIGDATAANALYTICSEMBEDDEDTHROUGHOUTOURPORTFOLIO

AGILEMANAGEMENT AGILEOPERATIONS DEVELOPERPRODUCTS

CONTINUOUSDELIVERY MAINFRAME SECURITY

APIanalyticsReleasemanagementdashboard AccessincidentresponseReal-timeappperformance

ConnectedIntelligenceCognitiveAdaptiveInsight(CAI)


Design QA/TESTDEV PRODUCTIONPRE-PROD

Specification

CI/Build Functionaltesting UATIntegration

testingPerformanceengineering

Deploytopre-prod

Codecommit SCM

DesignSpec

Requirements

BusinessUsers

TESTData

TESTStub

TESTData

User BA

ProductManager

CustomerUsers

CustomerExperience

CustomerExperience

CustomerExperience

CustomerExperienceDesign

BusinessValue

BusinessValue

BusinessValue

Improvement

Innovation

Intelligence

Assessment

Insight

Learning

Maintenance

Operations

Delivery

Testing

Support

Custom

er

DigitalAssurance

Value Insight

Learn Learn

Monitor Monitor TESTStub

REALUsers

TESTData

REALData

TESTStub

USERCase

Busine

ss

idea

Value-DrivenDelivery


DigitalAssurancePocketbook

PaulGerrard

NEW exclusiveCAsponsoredPocketbookauthoredbyPaulGerrard,eminentTest-Transformationconsultant,thoughtleader,teacherandinternationalconferencespeaker.Pre-orderyourFREE copytodayfromhttp://bit.ly/digital-assurance


CognitiveAdaptiveTesting(CA-T)Value-DrivenDelivery(VDD)

Lean&AgileTeams

Developers&Testers

ReleaseManagement

ProductOwner

Designer

ProductManager

DailyReviews

Roadmap

Vision

Backlog

SprintBacklogs

ValueDriven

PLAN

ShippableProducts

SHIFTLEFT

DESIGNOPS

SHIFTRIGHT

OPERATE

FeedbackLoops


Step1– ContinuousAdaptiveTestingIn-Sprinteverything

Sprint1 Sprint2 Sprint3 SprintN

Sprint2 Sprint3 SprintN

Sprint3 SprintN

UserStories

DevandManualTesting

AutomatedRegressionTesting

3weeks 3weeks 3weeks 3weeks

Sprint1

Sprint1 Sprint2

Release Release Release Release

Highlights

§ NoSprint-lageffect§ Releasein3weeks§ In-SprintAutomation


SoHowtoAchieveSpeed,QualityandCoverage?Pickanexistingapplicationandmodelrequirementsfor3Sprints

Write

CodeandTest

AutoRegression

Release

Model

CodeandTest

AutoRegression

Release

Model

CodeandTest

AutoRegression

Release

Model

CodeandTest

AutoRegression

Release

UserStories

DevandExploratoryManualTesting


3weeks 2weeks 2weeks 2weeks

AutomatedRelease

KeyPoints

§ SprintAcceleration§ Donotgo

Big-Bang§ Startsmall,see

results,thenscale§ Scalegraduallyin

theenterprise


Step2– LeanEngineeringPickanexistingapplicationandmodelrequirementsfor3Sprints

Model

CodeandTest

AutoRegression

Release

UserStories

DevandExploratory

ManualTesting


2weeks

AutomatedRelease

Sprint1 ReadUserStoryandcreatebasicModel

Automaticallygeneratebasicmanualtests

Exportthembackintoyourrepository(ALM,Rally,JIRA,

etc.)

Runtestcycleasusual

Sprint2ReadUserStoryandCreateModelwith

NegativeandAlternateScenarios

Automaticallygeneratemanualtestswithoptimal

coverage

Exportthembackintoyourrepository(ALM,Rally,JIRA,

etc.)

Runtestcycleasusual

Sprint3UserStoryisdirectlyModeled.Modelissinglesourceor

truth

Automaticallygeneratemanualandautotestswithoptimalcoverage

Automaticallyperformimpactanalysisonany

changes

Runtestcycleinacceleratedmode


ScalingModelingAcrosstheEnterpriseExampleofaRoadmapacrossBUs

BusinessUnit Project M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12

BU01

Project1 TransitiontoModelingcompletedafter3Sprintsand1extramonthofresults/benefitsanalysis,reviewand finalBUandPMOapproval

Project2 Modelingadoptioncompletedin2Sprints

Project3 Modelingadoptioncompletedin1Sprint

BU02

Project1 TransitiontoModelingcompleted,results/benefitsanalyzed,reviewedandapprovedbyendofSprint3

Project2 Modelingadoptioncompletedstillin3 Sprints

Project3


Scope

MappingCDValueStreamstoOrganizationLevel

Program/Portfolio

(CrossApps)

Project/Product

(CrossTeams)

Sprint(SingleTeam)

ValueStream

ContinuousStaging

andRelease

ContinuousTesting

ContinuousDevelopmentandIntegration

Epic

Feature01

US01

US02

US03

Feature02

US01

US02

US03

Feature03

US01

US02

US03


Persona-BasedJourneytoContinuousQuality

JediRank

Youngling

Padawan

Knight

Master

BusinessAnalyst

WritesRequirement

CreatesBasicModel

CreatesModelwithNegativeandAlternate

Scenarios

CreatesComplete,ComplexModels

Developer

WritesCodefromRequirements

WritesTestsFirst,thenWritesCodetoPassthe

Test(TDD).UsesContinuousIntegration

UsesServiceVirtualizationandAchieved

ContinuousTesting

LooksatEverythingasCode(EAC)andCodesInfraandEnvironment

Configurations.AchievedContinuousDelivery

DataScientist

ReadsReq.CreatesLotsofData

CreatesOnlyNeededDataBasedonModel-Specific

Paths

Auto-GeneratesStaticDataBasedontheModel

LinksTDMtotheModelforAutomated

ProvisioningofDynamicSyntheticData

APIEngineer

ReadsReq.ManuallyCreatesstatic

APIStubs

CreatesOnlyNeededR/RPairsBasedonModel-

SpecificPaths

Auto-GeneratesR/RPairsBasedontheModel

LinksVirtualServicestotheModelforAutomatedVirtualizationofAPIs

Tester

ReadsReq.CreatesManualTC

Auto-GeneratesManualTCsfromtheModelwith

BasicCoverage

Auto-GeneratesManualTCsfromtheModelwith

OptimalCoverage

LinksTestAutomationKeywordstotheModelforAutomatedTestScript

Generation

OperationsEngineer

CreatesanExtensiveRunbookinaWordDocumentandWorksSeparatelyFromthe

ProjectTeam

DropstheWordDocandUsesToolstoManageAppandEnvironmentConfig.StillNotPartoftheProjectTeamand

DoesManualDeployment

EnablesProperMonitoringcreatesAutomatedBuildand

DeploymentEnvironmentsforDev

Teams.PartoftheTeam

FocusesonContinuousDeliveryImprovements

whileDevelopersBecomeResponsibleforthe

ApplicationinProduction


CognitiveAdaptiveTesting

ExistingManualTestCases

DynamicOptimized–Tests

AgileRequirements

Designer

UnstructuredPartialcoverageTestbloatLoweffectivenessAnalog– difficulttomaintain

CognitiveAdaptive-Testsbasedon Insight

fromConnectedIntelligence

NewRequirementsandBusinessProcesses

Fewestteststeps(reduceby>30%)GreatercoverageHighereffectivenessFullydigitized– easytomaintain

ContinuousoptimizationHighesteffectivenessReduction>50%

Release OperationsDesign Development


ModelingMaturityAMBIVALENT

LEVEL1INITIALIZING

LEVEL2AWARELEVEL3

MINDFULLEVEL4

EXPERTLEVEL5

§ Testdataiscopiedfromproduction

§ ExtensiveuseofEndtoEndtesting

§ Testdataiscreatedmanuallybytesters

§ PIIintestdata,riskingbreachofsecurity

ChallengesofLevel1staywithfollowingimprovements:

§ Somedatabasesaresubsetted fortestingpurposedtospeedtestingandreducestoragecosts

§ PIIismaskedinSOMEtestdatabases

§ PIIremovedfromtestdatabases

ChallengesofLevel2reducewithfollowingdefinitivesteps:

§ Testdataiscreatedusingatool

§ PIIremovedfromtestdatabasessubsets

§ Testdataisexplicitlydrawnoutintestplans

§ TDMisrecognizedasaskill

Level3processesgetmatured:

§ Testdatacanbereserved

§ TestDatacoverageisunderstoodandknown

§ Therighttestsarerun,resultingincompletetesting

§ TestDataonDemandiswidespread.

§ CentralizedTDMteam

EverythinginLevel4+

§ EnterpriselevelModeling policyandstrategyfullyadoptedbythe organization

§ CommonEnterpriseModelRepositoryisleveragedusedbyallteams

§ ModelsarethesinglesourceoftruthforBiz/Dev/Test/Ops

§ ContinuousDevelopmentFrameworkinplace

MostITOrganizations


> COGNIT IVE ADAPT IVE <

RUNDEPLOYTESTBUILDPLAN

INTAKERequirementsUserStoriesReleasePlan

MODELImportUserStoriestoAutomaticallyCreate,VisualizeandOptimizeTests

InitiateFunctional, Non-Functional and SecurityasCode

FUNCTIONAL§ Subset/MaskTestData§ Create/ReserveTestData§ TestAutomationLibrary§ EnsureMobileExperience

INSIGHT-DRIVEN

CONFIG/DEPLOY§ ProvisionEntireStack§ ConfirmConfigurations§ ApproveChanges§ SuccessfullyDeploy§ InternalorExternalCloud

INTEGRATIONRemoveConstraintswithVirtualServices§ Mobile,Web,AppServer,

Middleware,Backend,MF§ 3rd PartySystems/API’s

CODEDevelopandCommitCode,ScanCode,VersionControl,ContinuousIntegration.CompleteBuildandInitiateRelease

AUTOMATION AUTOMATION AUTOMATION

ü Speed/Time-to-Marketü Quality/Availability

AUTOMATION

ü Cost/FinancialMixü Risk/Compliance

ü Requirementsü Environments

ü Dataü Automation

CustomerExperience

NON-FUNCTIONAL§ PerformanceEngineering§ SimulateBackendLoad§ TestOutlierConditions§ EnsureMobileExperience§ Security/Penetration

CommonGoals “BigPicture”Issues

DESIGN SHIFT-LEFT SHIFT-RIGHT OPS

Node.Probe

Node.TestTDDBDD

ATDD

BPMNNimbus

Visio

Node.Data

Node.Learn

MEASURE/FEEDBACK§ CustomerExperience§ BusinessServiceView§ ApplicationView§ InfrastructureView§ DynamicCapacity§ Feedback loop


DESIGNOPS

VALUE-DRIVENDELIVERY

CognitiveAdaptiveAdoption

1CognitiveAdaptiveLearning

2CognitiveAdaptiveTesting

3CognitiveAdaptiveTechnology

4CognitiveAdaptiveDelivery


CognitiveAdaptiveAdoption(CA-A)ThefutureisValue-DrivenDelivery

CAProject&PortfolioManagement

Unifylongtermstrategy,investmentandportfolioplanning.

CAAgileCentral

Collaborativelyplan,prioritizeandtrackworkacrosstheenterprise.

Develop&Test

Synchronizeeffortstodramaticallyspeedappdevelopment&increasequality

Manage&Monitor

Makeagreatcustomerexperienceacompetitiveadvantage

Release&Deploy

Controlthereleaseprocess,tocontinuouslyadvanceapplicationquality,improvethecustomerexperienceandreducecosts.

MinimumViableDelivery

MinimumViableExperience

MinimumViableQuality

Value-DrivenDelivery(MinimumViable-Value-Validated)

ExecutivesBUSINESSINITIATIVE

Complexity-InformedOrganizationalChange

TheDigitalEcology

MinimumViableProduct

CUSTOMER

THINK

MAKE


Q&A


Whatnext?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5X29SModel-BasedTestingforAchievingMaximumCDPipelineAcceleration

11/16/2016at3:00pm

DO5X47S 'Shift-Right'- RapidEvolutionwithDesignOps 11/16/2016at3.45pm

DO5X40S CaseStudy:WhyRabobankWasInspired 11/17/2016at3.45pm


MustSeeDemos

UseModelBasedTesting

CAARDDevOpsTheatre5

DevOps5Theatre

AchieveMaxTestCoverage

CAARDDevOpsTheatre5

ImproveDataCompliance

CATestDataManagerDevOpsTheatre5

DeliverTestDataFaster

CATestDataManagerDevOpsTheatre5


Stayconnectedatcommunities.ca.com

Thankyou.

[email protected]

JonathonWright

@Jonathon_Wright

SlideShare.net/Jonathon_Wright

LinkedIn.com/in/Automation

https://www.ca.com/us/trials/ca-agile-requirements-designer.register.html

How to Solve Problems of Testing in Agile Environments

Technology

Transcript of How to Solve Problems of Testing in Agile Environments