Stabilizing Your Continuous Testing Suites-DevOps …...3 Eran!Kinsbruner!!...

AT4 Agile and Continuous Testing Thursday, November 7th, 2019 10:00 AM

Stabilizing Continuous Testing in DevOps

Presented by:

Eran Kinsbruner

Perfecto

Brought to you by:

888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] https://agiledevopseast.techwell.com/

Eran Kinsbruner Eran Kinsbruner is the chief evangelist and author at Perfecto, a Perforce company. He authored two books, The Digital Quality Handbook and Continuous Testing for DevOps Professionals. Eran is also a monthly columnist at InfoWorld.com and The Enterprisers Project. Eran is a software engineering professional with nearly twenty years of experience at companies such as Matrix, Sun Microsystems, General Electric, Texas Instruments, and NeuStar. He holds various industry certifications from ISTQB, CMMI, and others. Eran is a recognized mobile testing influencer and thought leader, as well as an experienced speaker in the major software engineering conferences. He is also a public speaker, researcher, blogger, and a patent-‐holding inventor for a test exclusion automated mechanism for mobile J2ME testing. He can be found all over social media, including on Facebook, Twitter (@ek121268), LinkedIn, and his professional blog, continuoustesting.blog.

10/29/19

1

1 | Stabilizing Con3nuous Tes3ng in DevOps perfecto.io

Stabilizing Con-nuous Tes-ng in DevOps

2 | Agile vs. DevOps for Con-nuous Tes-ng: How to Op-mize Your Pipeline perfecto.io

•  Chief Evangelist and author at Perfecto

•  Blogger and speaker

•  19+ years in development & tes3ng

•  Author of “The Digital Quality Handbook” and “Con3nuous Tes3ng for DevOps Professionals”

•  TwiOer: @ek121268

•  Blog: con3nuoustes3ng.blog

•  Website: perfecto.io

Today’s Speaker: Eran Kinsbruner

10/29/19

2


Today’s Agenda

1

2

3

Agile Tes3ng Manifesto

How to Decide Which Tests to Automate

Test Case Cer3fica3ons in Con3nuous Tes3ng

4 Test Maintenance and Test Aging

5 Live Demo


The Agile Tes-ng Manifesto T H E D E V O P S M A N I F E S T O

ü  Con;nuous tes;ng over tes3ng at the end. ü  Embracing all tes;ng ac;vi;es over only automated func3onal tes3ng.

ü  Tes3ng what gives value over tes3ng everything. ü  Tes;ng across the team over tes3ng in siloed tes3ng departments.

ü  Product coverage over code coverage.

10/29/19

3


ShiK LeK: Manual vs. Automated Tes-ng Impact


Are You Measuring Your DRE?

DRE (Defect Removal Efficiency) = Defects removed during the development phase

Defects detected later in the cycle (UAT, Produc3on)

Coverage, lack of and late automa3on tes3ng, designed for testability, unit tes3ng, outdated environments/plaborms.

10/29/19

4


1.  What’s the test engineer’s gut feeling 😊

2.   Risk calculated as probability to occur and impact to customers.

3.   Value — Does the test provide new informa3on and, if failed, how much 3me to fix?

4.   Cost efficiency to develop — How long does it take to develop and how easy is it to script?

5.   History of test — Volume of historical failures in related areas and frequency of breaks.

Source: Angie Jones

Test Automa-on – What to Automate?

S C O R E S : 67-‐100 = Automate | 34-‐66 = Possibly Automate | 0-‐33 = Don’t Automate


Cer-fying Test Automa-on

10/29/19

5


Test Automa-on Cer-fica-on

•  Think con3nuous value. •  Think maintenance and complexity to create, execute, monitor, and maintain (ROI).

•  Test reliability over 3me.

If { (Test Scenario) == Candidate to add Value add { and x– It detect defects} { and – It passes across mul3ple plaborms with consistent result} { and – Its execu3on 3me is < 5 minutes }

Then { Include in Test Suite & & CI } Else {

}


GeUng Started With Con-nuous Tes-ng (Decision Making Process)

•  Test Objec;ves •  What should it cover? Not overly complicated things. •  Should it be part of each build-‐acceptance/CI job? (Keep the pipeline GREEN) •  Reliability when running across mul3ple plaborms/OS versions? •  Time to develop the test and maintain it? •  Coverage area should be top of mind.

•  Test development early warnings (mark tests as “flaky”).

10/29/19

6


Personas and Tools Balance


I N T E R A C T I V E T E S T S

UI/UX manual tests

Balancing Test Crea-on for the Three Different Personas With the Right Tools

O P T I M I Z E D M O D E L

Developers & SDETs (Code-‐Based)

Business Testers Ownership (Codeless)

Business Testers Ownership

10/29/19

7


Collabora-ve Approach — Codeless & Code-‐Based

14 | Quest 2019 perfecto.io

Tool/Framework Selec-on Criteria: Technical Fit & Skills

SDLC process fit (integra3on, plug-‐ins, skills,

etc.)

Community size, support, and docs

Feedback loop and repor3ng

Automa3on coverage

Cloud and automa3on at

scale

Automa3on robustness and maintainability

10/29/19

8


Web Tes-ng Landscape – Comparison Case Study

16 | Stabilizing Con-nuous Tes-ng in DevOps perfecto.io

Cross-‐Browser Tes-ng Tools (Open-‐Source) — Download Trends Vary

Protractor

Puppeteer

WebDriver IO

Cypress IO

Selenium WebDriver

Source: npmtrends

10/29/19

9


Addi-onal Categoriza-on of JavaScript Browser Tes-ng Tools

Tools that... •  Provide a tes3ng structure (BDD/ATDD) (Mocha, Jasmine, Jest, Cucumber).

•  Provide asser3on func3ons (Chai, Jasmine, Jest, Unexpected).

•  Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma).

•  Generate code coverage reports (Istanbul, Jest, Blanket).

•  Provide a browser or browser-‐like environment with control of scenario execu3on, UI tes3ng, and more (Protractor, Nightwatch, Phantom, Casper, Selenium, WebDriver.IO, TestCafe).

•  Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble).

Source: Medium.com


High-‐Level Comparison Between Tools and Prac-ces Test Framework

Supported Dev Languages

Supported Browsers

Supported Test Frameworks

Setup and Execu;on

Integra;ons Breadth of Tes;ng Op;ons

Maturity, Documenta;on, Support

Required Skillset Cloud and Execu;on at Scale

Selenium WebDriver

Java, C#, Java Script, Python, Ruby, Objec3ve-‐C

Chrome, Safari, Firefox, Edge, IE

Mocha JS, Jest, other super set on top of Selenium (Protractor, WebDriverIO, etc.

Download relevant driver, set up a grid, network and loca3on impacts execu3on speed

Plenty of integra3ons (CI, CD, repor3ng, visual tes3ng, cloud vendors)

End-‐to-‐end, security, unit,

Robust community, mul3ple bindings, best prac3ces

Coding skills required Perfecto fully supports Selenium and its WebDriver configura3ons. Local execu3on requires sezng up a Selenium grid

Cypress.io JavaScript Chrome, Electron

Mocha JS Comes with bundled chrome browser, no complex environment setup

Limited integra3ons

End-‐to-‐end

Good documenta3on and code sample, growing community

Coding skills required Built-‐in Chrome browser, limited

Codeless Irrelevant, based on record and playback

All Proprietary UI with underlying Selenium WebDriver APIs

Mostly SaaS/browser plugin installa3on

Limited Func3onal/UI

Growing, limited No coding skills required

Perfecto supports codeless in the cloud

BDD Java, Ruby, JS, Kotlin

All Junit, Selenium, Appium

Open-‐Source, Maven/Gradle/TestNG

Plenty + APIs (e.g. Rest Assured)

Func3onal Robust community, docs, adop3on

Step-‐defini3on development in code is required/scenarios are no-‐code

Perfecto Quantum is a web/mobile BDD framework

10/29/19

10


How to Measure Test Value Through Repor-ng and Analy-cs


Error Classifica-on and Test Produc-vity

10/29/19

11


The Power of Test Data Analysis

1.  Long history of failures A.  Test vs. device issue B.  Exclude and debug

2.  Script development prac3ces A.  Popup handling B.  Element not found

3.  Are the tests bring any value?

4.  Plaborm Health?


Slicing and Dicing Test Data

10/29/19

12


How Do Tests Perform Within CI?


Object Iden-fica-on Weights Based on History

10/29/19

13


The 4 Key Pillars of Con-nuous Tes-ng in DevOps

ANALYSIS CREATION EXECUTION LAB

Unified solu-on serves DevOps with greater efficiency.

Lab for con3nuous tes3ng •  Backbone for all tes3ng

ac3vi3es •  Hos3ng of devices,

management, & logis3cs •  Availability & scaling •  Suppor3ng func3ons

Test execu3on within the pipeline •  Scale •  Orchestra3on •  Frameworks

Test crea3on and maintenance •  Authoring tool •  Valida3ons

Test analysis •  Fast feedback •  Root cause analysis


LIVE DEMO

•  Execu3on of BDD test code in parallel. •  Execu3on of codeless Selenium in parallel. •  ML/AI in ac3on (object locators & test analy3cs).

L I V E D E M O

10/29/19

14


Measuring Con-nuous Tes-ng Success in DevOps


Some Recommended CT Measurements (KPIs)

•  How fast are tes;ng ac;vi;es moving, and what is slowing down these ac3vi3es? •  Test flakiness. •  Test dura3on. •  Percentage of automated vs. manual tests.

•  Applica;on quality measurements. •  Number of escaped defects and in which areas. •  MTTD — mean 3me to detec3on of defect. •  Build quality.

•  Pipeline efficiency measurements. •  Number of user stories implemented per itera3on. •  Test automa3on as part of DoD across itera3ons. •  Broken builds with categories. •  CI length trending. •  Lab availability and u3liza3on.

•  Quality cost measurements. •  Opera3onal costs, lab availability issues. •  Cost of hardware/so|ware. •  Costs of defects by severity and stage

10/29/19

15


Perfecto’s Smart Con-nuous Tes-ng Solu-on

Smart Execu;on Fast and parallel test execu3on with mul3-‐team orchestra3on abili3es and management.

Smart Analy;cs "Single pane of glass" provides visibility and scales to support millions of test results.

Smart Crea;on Automa3on crea3on that matches your

team’s skillset (Appium, Espresso, XCUITest, Quantum BDD, Codeless).

Smart Lab Always on and stable. Always up to date. Supports all mobile OS

and plaborms.

The Perfecto human factor increases your

chances to succeed.

Black Belt Tes;ng Experts Training Dedicated

Success Manager 24/7 VIP Support


10/29/19

16


Thank You

Stabilizing Your Continuous Testing Suites-DevOps …...3 Eran!Kinsbruner!!...

Documents

Transcript of Stabilizing Your Continuous Testing Suites-DevOps …...3 Eran!Kinsbruner!!...