Agile vs Fragile - ANZTB · Susmit Bhattacharya, Solution Architect, Asia Pacific Agile vs Fragile...

34
Susmit Bhattacharya, Solution Architect, Asia Pacific Agile vs Fragile - The need for Automation in Agile © 2017 Tricentis GmbH. All Rights Reserved.

Transcript of Agile vs Fragile - ANZTB · Susmit Bhattacharya, Solution Architect, Asia Pacific Agile vs Fragile...

Susmit Bhattacharya, Solution Architect, Asia Pacific

Agile vs Fragile- The need for Automation in Agile

© 2017 Tricentis GmbH. All Rights Reserved.

© 2017 Tricentis GmbH. All Rights Reserved. 2

Critical Physical Limit

Ever-Growing Challenge

Technical Complexity

& Rate of Change

YearsMonths

MonthsWeeks

WeeksDays

DaysHours

HoursMinutes

Delivery

Cycle Time

Perceived DisruptionBeyond

Continuous

Testing

Today

© 2017 Tricentis GmbH. All Rights Reserved. 3

Requirements Design Implementation Testing Acceptance Deployment

Inception Elaboration Construction Transition

Development Operations

DevOps

Agile Event Horizon

Digital Singularity

IncrementalRational Unified Process

4

AgileScrum, Kanban

2

ContinuousDevOps

1

6+

Big BangWaterfall

Development Response = wall of confusion

Flexible Framework

Rigid Rules

Following a Plan

Contract Negotiation

Copious Documentation

Processes & Tools

Individuals & Interactions

Working Software

Customer Collaboration

Responding to Change

© 2017 Tricentis GmbH. All Rights Reserved. 4

Dev Ops

Continuous

Release & DeploymentCollaborative

Development

Continuous

Testing

Continuous

Monitoring

Continuous

Integration

Continuous

Feedback

DevOps Movement

DevOps is no longer a

unicorn’s discipline.

TestDev OpsShift Left Shift Right

of enterprises have

adopted or plan to adopt

81%

DevOps.

© 2017 Tricentis GmbH. All Rights Reserved. 5

Current State Target State

remaining manual tests(exploratory and manual spec-based)

automated UI tests

automated API tests

+

service virtualization(key enabler for high automation)

GUI

API

20-40%

60-80%Specification

driven

5-20%

80-95%

Testing Future

Budgets won’t increase

© 2017 Tricentis GmbH. All Rights Reserved. 6

remaining manual tests(exploratory and manual spec-based)

automated UI tests

automated API tests

+

service virtualization(key enabler for high automation)

GUI

API

Model-Based Test Automation (MBTA) for

UI and APIAutomating the automation framework.Automate

Session-Based Exploratory TestingMake your testing intellectually rich.

Explore

Orchestrated Service VirtualizationMake testing independent.

Stateful Test Data Management Manage

Computer-Aided Test Portfolio DesignRedundancy-free test case portfolio with

maximum risk coverage and fewer tests.Optimize

Integrate

CT Integration (CI, CD)

Distributed ExecutionTarget State

Testing Future

Current State

Budgets won’t increase

© 2017 Tricentis GmbH. All Rights Reserved. 7

remaining manual tests(exploratory and manual spec-based)

automated UI tests

automated API tests

+

service virtualization(key enabler for high automation)

GUI

API

Computer-Aided Test Portfolio DesignRedundancy-free test case portfolio with

maximum risk coverage and fewer tests.Optimize

Target State

Testing Future

Current State

Budgets won’t increase

© 2017 Tricentis GmbH. All Rights Reserved. 8

Low

Risk

High

Risk

Medium

Risk80%20%

Business RiskCoverage

TestCases

The time needed for

testing is infinitely larger

than the time available.

Critical Limit

Risk-Based Testing

Have the right test cases.

Do the right things.

© 2017 Tricentis GmbH. All Rights Reserved. 9

risk

co

vera

ge

60%

80%

100%

40%

20%

0%

𝑴

𝑵=

0

defe

ct r

ate

max

100%

test cases

20%0% 40% 60% 80%

critical defects

M N

𝟏 weight𝟏 𝟏 𝟏 𝟏 𝟏 𝟏… …[%]

Computer-Aided Test Portfolio Optimisation

© 2017 Tricentis GmbH. All Rights Reserved. 10

60%

80%

100%

40%

20%

0%

risk

co

vera

ge

100%20%0% 40% 60% 80%

test cases

critical defects

0

defe

ct r

ate

max

M N

𝒘𝟏 𝒘𝟐 𝒘𝟑 𝒘𝟒 𝒘𝒊 𝒘𝑵… … … … 𝒘𝒊 ≥ 𝒘𝒊+𝟏weight

Computer-Aided Test Portfolio Optimisation

© 2017 Tricentis GmbH. All Rights Reserved. 11

60%

80%

100%

40%

20%

0%

IntuitiveTest Design

MethodicalTest Design

100%20%0% 40% 60% 80%Test Cases

Risk

Coverage

Risk-Based Testing

Achieve maximum risk coverage with

a minimal number of powerful test cases.

© 2017 Tricentis GmbH. All Rights Reserved. 12

remaining manual tests(exploratory and manual spec-based)

automated UI tests

automated API tests

+

service virtualization(key enabler for high automation)

GUI

API

Model-Based Test Automation (MBTA) for

UI and APIAutomating the automation framework.Automate

Optimize

Target State

Testing Future

Current State

Budgets won’t increase

© 2017 Tricentis GmbH. All Rights Reserved. 13

2nd Generation

Test Automation

Framework

1st Generation

Record

Replay

10%Automation

Evolution of Automated Testing

© 2017 Tricentis GmbH. All Rights Reserved. 14

Example:

“…select invoice with the car registration number VIC-3456”

Great for developers who can code

...but is even harder to understand for the business team

Conclusion: Only for developer & Expensive to maintain

Keyword Driven Data Driven

Custom Hybrid

Functionally DecomposedCommercial

Script Based Test Automation Frameworks

© 2017 Tricentis GmbH. All Rights Reserved. 15

Script-Based

2nd Generation

Test Automation

Framework

1st Generation

Record

Replay

30%Automation

10%Automation

+ -Programming Required

- +Business Readable

Model-BasedTest Automation

Next Generation

NoScripting

90%Automation

+

Future of Automated Testing

© 2017 Tricentis GmbH. All Rights Reserved. 16

Technical Information

Application Pages

sampleapp.tricentis.com

Technical Name Value Range

-

-

1;2;3;4;5;6;7;8;9

Petrol;Diesel;Gas;Other

-

-

{CLICK};{RIGHTCLICK}

Private;Commercial

2016;2015;2014;2013

-

Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Next

Usage

Year of Construction

License Plate Number

Insurant Data

Product Data

Next {CLICK}

Start Date 01/03/2016;01/04/2016

Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio

Payment Option Yearly;Quarterly;Monthly

Next >>

Make Audi

Engine Performance [kW] 200

Number of Seats 5

Fuel Petrol

Year of Construction 2016

List Price [$] 35.000

License Plate Number B-CD 123

Annual Mileage [mi] 10.000

Usage Private

Scan Application…. Silver; Gold; Platinum; Ultimate

Quote Details

Price per Year($);Online Claim; ..

© 2017 Tricentis GmbH. All Rights Reserved. 17

Logical Name

Enter Insurant Data

Create Vehicle Insurance

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Next

Usage

Year of Construction

License Plate Number

Value

Audi

200

5

Petrol

35.000

10.000

{CLICK}

Private

{Year}

B-CD 123

Action

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Enter Product Data

Next

Start Date

Insurance Sum [$]

Payment Option

{CLICK}

01/03/2016

7Mio

Yearly

Input

Input

Input

Input

Verify Quote Details

Price List

Price per Year($)

Silver 1.535,22 Verify

Select

Select

Business Information

Technical Name Value Range

Technical Information

-

-

1;2;3;4;5;6;7;8;9

Petrol;Diesel;Gas;Other

-

-

{CLICK};{RIGHTCLICK}

Private;Commercial

2015;2014;2013;2012

-

Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Next

Usage

Year of Construction

License Plate Number

Insurant Data

Product Data

Next {CLICK}

Start Date 01/03/2016;01/04/2016

Insurance Sum [$] 3Mio;7Mio;10Mio;15Mio

Payment Option Yearly;Quarterly;Monthly

Quote Details

Silver; Gold; Platinum; Ultimate

Price per Year($);Online Claim; ..

It’s like playing Lego.

© 2017 Tricentis GmbH. All Rights Reserved. 18

100%

0%

Sprint

Start

Sprint

End

API

UI

Start testing early.

Shift left test automation.

API Testing

Degree of

Completion

Time

Efficiency

Gain

+4xCreation

+6xMaintenance

+20xExecution Future test automation must focus on API testing.

© 2017 Tricentis GmbH. All Rights Reserved. 19

Logical Name

Enter Product Data

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Next

Usage

Year of Construction

License Plate Number

Enter Insurant Data

Verify Quote Details

Next

Start Date

Insurance Sum [$]

Payment Option

Create Vehicle Insurance

Price List

Insurance Sum

Gross Premium [$]

Value

Audi

200

5

Petrol

35.000

10.000

{CLICK}

Private

{Year}

B-CD 123

{CLICK}

01/03/2016

7Mio

Yearly

1.535,22

Action

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information

UI Test CaseBrowser

Audi

200

5

Petrol

35.000

10.000

Private

{Year}

B-CD 123

01/03/2016

7Mio

Yearly

1.535,22

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information

API Test CaseWebservice

Enter Product Data

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Usage

Year of Construction

License Plate Number

Enter Insurant Data

Verify Quote Details

Start Date

Insurance Sum [$]

Payment Option

Price List

Insurance Sum

Gross Premium [$]

© 2017 Tricentis GmbH. All Rights Reserved. 20

Logical Name

Enter Product Data

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Next

Usage

Year of Construction

License Plate Number

Enter Insurant Data

Verify Quote Details

Next

Start Date

Insurance Sum [$]

Payment Option

Create Vehicle Insurance

Price List

Insurance Sum

Gross Premium [$]

Value

Audi

200

5

Petrol

35.000

10.000

{CLICK}

Private

{Year}

B-CD 123

{CLICK}

01/03/2016

7Mio

Yearly

1.535,22

Action

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information

UI Test CaseBrowser

Audi

200

5

Petrol

35.000

10.000

Private

{Year}

B-CD 123

01/03/2016

7Mio

Yearly

1.535,22

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Input

Verify

Select

Select

Business Information

API Test CaseWebservice

Enter Product Data

Enter Vehicle Data

Make

Engine Performance [kW]

Number of Seats

Fuel

List Price [$]

Annual Mileage

Usage

Year of Construction

License Plate Number

Enter Insurant Data

Verify Quote Details

Start Date

Insurance Sum [$]

Payment Option

Price List

Insurance Sum

Gross Premium [$]

Manual Testers become Automation Specialists

© 2017 Tricentis GmbH. All Rights Reserved. 21TDD vs BDD.

Behaviour-Driven Development (BDD) is a software development process based on Test-Driven Development (TDD)

BDD combines techniques and principles of

• TDD• Domain-Driven Design• Object-Oriented Analysis & Design

BDD gives a clearer understanding as to what the system should do from the perspective of the developer and the customer.

TDD only gives an understanding of what the system should do from a developer perspective.

© 2017 Tricentis GmbH. All Rights Reserved. 22

Better collaboration between developerstest analysts and business analysts

Executable documentation of the software.

BDD/TDD – The promise.

© 2017 Tricentis GmbH. All Rights Reserved. 23

Open Source (Cucumber, JBehave, Specfolw with Selenium, Watir, etc.) • Requires coding! Each GIVEN, WHEN, THEN, AND in the BDD scenario (left) needs a matching function call (right).• No support for systematic scenario identification and definition

BDD – A Working Example.

© 2017 Tricentis GmbH. All Rights Reserved. 24BDD using TCD.

© 2017 Tricentis GmbH. All Rights Reserved. 25

remaining manual tests(exploratory and manual spec-based)

automated UI tests

automated API tests

+

service virtualization(key enabler for high automation)

GUI

API

Session-Based Exploratory TestingMake your testing intellectually rich.

Explore

Optimize

Target State

Testing Future

Current State

Budgets won’t increase

© 2017 Tricentis GmbH. All Rights Reserved. 26

Formal TestingInformal Testing

Exploratory TestingManual

Testing

Automated

Testing

Ad-hoc

Testing

Testing Continuum

Testing Continuum

© 2017 Tricentis GmbH. All Rights Reserved. 27

Risks

Automated Testing

I’m an automated test case.

Usability Issue

Stability Issue

Reliability IssueScalability Issue

Coherence Issue

Understandability Issue

Testability Issue

Convenience Issue

Security Issue

Accessibility Issue

Performance Issue

Functional Issue

Automation is doing, what automation does.

© 2017 Tricentis GmbH. All Rights Reserved. 28

1 2 3 4 5 … … … … … … …

Risks

Automated Testing

© 2017 Tricentis GmbH. All Rights Reserved. 29

I’m an automated test case.

Manual Testing

I’m the same test case executed manually.

© 2017 Tricentis GmbH. All Rights Reserved. 30

Plan as you test!

Story-Based

Motivating

Credible

Exploratory Testing

Exploratory BranchingThe art of deciding on what to test next.

NewTesting

Idea Exploratory TestingPerfect counterpart to specification-based

automated & manual testing.

Test Cases vs. ScenariosPre-specified inputs vs. hypothetical situation

© 2017 Tricentis GmbH. All Rights Reserved. 31

Specification

Based Testing

Exploratory Testing

Main Purpose

Analyze Potential RisksProblem vs. No Problem

Monitor Known RisksPass Result vs. Fail Result

*That’s an illustration. Don’t confuse it with reality.

Mechanical Testing vs. Intelligent Testing≫ Testing as artifact creation vs. testing as performance ≪

© 2017 Tricentis GmbH. All Rights Reserved. 32

Specification

Based Testing

Exploratory Testing

Analyze Potential RisksProblem vs. No Problem

Monitor Known RisksPass Result vs. Fail Result

Agile Testing Law

Checked≫ Efficient Automated Testing ≪

Explored≫ Effective Exploratory Testing ≪

Tested≫ Productive Testing ≪

+

=

𝑅𝜇𝜈 −1

2𝑅𝑔𝜇𝜈 + Λ𝑔𝜇𝜈 =

8𝜋𝐺

𝑐4 𝑇𝜇𝜈

Agile Equation Of Motion

© 2017 Tricentis GmbH. All Rights Reserved. 33Core Conclusion

Automation is a must.

Manual testing is unjustifiable.

Exploratory testing is a necessity.

*We don’t need humans doing something that a machine can do. What we do want the human testers doing is exploratory testing.

We are Passionate about Testing„ „