Quality Engineering in DevOps world a Strategic...

www.cigniti.com | Unsolicited Distribution is Restricted. Copyright © 2015 - 16, Cigniti Technologies

Quality Engineering in DevOps world – a Strategic Enabler

www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 2

» Analyst Speak

» DevOps in a nutshell

» DevOps vs DevTestOps

» Continuous Testing & Delivery leveraging QE

» Unit Testing & SAST

» Automation , Performance & Security

» Service Virtualization

» Continuous Integration , Deployments & Delivery

» Operations Stabilization in Devops

» Continuous Automation & Delivery Pipeline

» Collaborative Platform & How does it help?

Agenda


Analyst Speak

To date, Gartner has had more than 3,500 client interactions (inquiries, best-practice calls and conference one-on-one meetings) discussing DevOps awareness, initiatives, successes and failures

Forrester - Development and operations (DevOps), the purview of unicorns, is now transitioning to the mainstream. Business technology (BT) leaders are realizing that Agile is no longer enough; they need to involve the full technology management organization to drive competitiveness in an era when customer centricity is pivotal to success


Analyst Speak


Key Challenges

DevOps toolchains are often built from discrete and sometimes disconnected tools, making it difficult to understand where bottlenecks are in the application delivery pipeline

DevOps initiatives continue to grow in complexity, driving many IT organizations to seek tools that provide greater control and visibility over the entire DevOps process

Toolchain orchestration promises to simplify the DevOps toolchain; however, one tool is not able to support the entire toolchain resulting in additional integration, ownership and handoff challenges when using multiple orchestration tools

Orchestrating the DevOps toolchain can be compelling, but it can introduce another set of challenges

01

02 03

04


DevOps adoption Is no longer Just For unicorns

Source: Forrester’s Q1 2017 Global DevOps Benchmark Online Survey Source: Forrester Data Global Business Technographics Developer Survey, 2016 & Forrester Data Global Business Technographics Infrastructure Survey, 2016

Culture and people come First;

process follows

Release velocity is pivotal There’s no “easy” button


DevOps Adoption


Adoption and Strategies vary across Vertical Segments

$

Business Services And construction Ponder The Leap Into Automation

Media, entertainment, & Leisure embark On The Journey To Differentiate With DevOps

Public Sector And Healthcare cling To Manual Audit And compliance Procedures

Retail And Wholesale DevOps Adoption Drive Differentiated Interactions With clients

Compliance-wary companies need not fear automation

Utilities And Telecommunications Lead The Pack To Drive Business Transformation

Data must drive strategy, with services that emphasize connectivityFinancial Services Accelerate To compete

With FintechsDevOps offers key benefits in fraud detection, time-to-market, and scalability

Get ready for the next generation of digital stores

DevOps drives to improve customer experience

Manufacturing Transitions Beyond Automation To Show DevOps Momentum

Manufacturers improve efficiency and insight by expanding the role of technology

Technology is becoming part of the business


DevOps vs. DevTestOpsD e v O p s

End to End Automation

Overall Cost Reduction – Automation, Reduce Cost of Quality

Fail – First, Fail - Fast, Fail- Often – Early Defects Detection

Collaborative Communication utilizing Dashboard

Continuous Testing

D e v Te st O p s

Equal Importance for Dev, Operations and Testing

All actions – Development --> Operations

Emphasis on Automation & Analytics

Continuous Integration + Continuous Deployment (CI/CD)

Dev --> Ops Process Reduced from Months to Days/Hours

Continuous Feedback & Improvement Supported

Maturing Processes and Organizations Focus


Devops in a nutshell

Build Dev

Automated Unit Testing & SAST

Build Packing & Automated

Deployments

Continuous Testing

• Functional Automation• Non Functional Automation

Production Deployments

Post Production Monitoring & Feedback

Amplification

Virtual Server(s)

Physical server(s)

CI Server

Hosted Tools

Infrastructure Automation &

operations


Continuous Testing & Delivery Leveraging QE elements

Client Layer Web Server App Server Component 1 Component 2 Component 3

Target System

Target System

Service Virtualization

Data Engineering Dashboard

Application Under Test

CI Platform

UI

Services

App Component

Continuous Monitoring

Health Check

InfrastructureApplication {Through scripts} {Through metrics configuration}

Functional Automation

Continuous Testing

Non - Functional Automation

Leveraging a collaborative dashboard to monitor application release activities & health

Enabling continuous testing by overcoming the dependency factors involved

Centralised data management through automated TDM tools & other mechanisms for on demand testing

Ensuring faster feedback for issues remediation through continuous infrastructure & application monitoring in production & test environments

Adopting Quality Engineering approach to provide end to end automated functional and non – functional test coverage for each release

Ensuring quality during rapid and frequent build deployments by leveraging robust CI tools integrated with test automation frameworks

Third party system

Efficient Tools

Leve

rag

e


DevOps Journey

DevOps Journey Automation Maturity

Partial Agile No Automation in place

Functional and NonFunctional Automation

Adopted CI and CD tools

Implemented Dual Shift Approach

Build - Deployment –Release Automation

Matured Agile Organization

Transformed to TDD / ATDD / BDD

DevOps

Matured Collaborative Agile ( Dev, QA and Ops)

Cost Elements ROI Factors

» Tools for Automation and Non Functional Tests

» Process Standardization

» On Demand Test Envts.» End-to-End Automation

and Test Coverage

» Tools and process for CI and CD

» Infra Maintenance

» DevOps Process» Org. Change Mgmt.

(People, Process and Systems)

» Cost of Maintenance » Technology Adoption

» Unified Automation Framework

» Best suited tools

» Tool optimization and consolidation

» Quality Check gates

» Automation Stabilization » Continuous Testing –

Continuous Feedback

Quality Engineering approach for early testing (Functional and Non Functional)

» Build Reliability» Uninterrupted Service

Delivery


In DetailKey QE elements


Unit Testing & SAST

JUNIT/NUNIT

Static Code Analysis

Build Server2

2

CI Server

Unit Testing

1

Open Source/Commercial Tools

3

Results Analysis & Vulnerable

identification

Manual Analysis for false positive

removal

Final Summary

Code Commit

1

I D E

(Technology Agnostic)

Code Repository

» Unit testing is performed using respective open source tools that integrated with the IDE used for code development

» CI Server picks the build from the code repository/version controlling tool once the developer commits the build

» CI Server initiates the static code analysis by triggering a request to the configured open source / commercial tools

» Input Validation » Output Encoding » Secure handling of credentials » Session Management » Cryptographic practices

» Error handling / Logging» Data Protection » Communication Security » Memory management

Secure Coding Guidelines & Standards


Automation, Performance & Security

CI PlatformBuildVersion

Controlling Tool

Test Automation Performance Testing

Security Testing (DAST)

Test Automation Tools

Test Management ToolsTest Automation

Framework

Performance Testing Tools

Performance Testing Scenarios

Monitoring Tools

Trigger Monitoring High Level Analysis Report

Trigger Automated Test Execution

DAST Tools

Monitoring Tools

Results Analysis & Vulnerabilities Identification

Manual Analysis (False Positives)

Automated Trigger

Final Summary Report


TDD blended with DevOps

Function 1- Unit Test case

Function 2- Unit Test case

Function 3 - Unit Test case

Feature broken into functions

Add a Developertest

Run Developertests at the unit level

Make code changesRun the Developer testsPass,

Functionality completeDevelopment Ends

Fail

Fail

Pass

Pass,Functionality Incomplete

Fin

aliz

ed

Fe

atu

re

Unit Testing, Integration Tests and Performance

Tests in reference to development under DevOps practices.

DevOps Specific input to feature

requirements

Feature Spec file

Automated Configuration Management frameworks

(Ansible/Chef/Puppet)

CI/CD framework(Continuous Integration/

Continuous Delivery)

TD

D

De

vO

ps

Co

ntin

uo

us

Fee

db

ack

DevOps Analytics PlatformContinuous Deployment


Service Virtualization in AWS Cloud

T

Legacy

ERP

Databases

Mainframe

» Automatically Capture realistic behavior

» Build VS Image, collection services requests and Service responses

» Optimize models as desired

Create Virtual Services

Playback against Virtual Service

Test (Manual, Automated)

System Under Test (SUT)

Virtual Service

SOAP Request

SOAP Response

JMS Publish

Exercise SUT

Serv

ice

Req

ues

t

Serv

ice

Res

po

nse

Service Response

Service Request

Service Response

Service Request

Serv

ice

Res

po

nse

Serv

ice

Req

ues

tDevTest

Deploy SV tool on cloud

Advantages

✓ Accessible from Multiple Geographies (Cloud & On-premise)

✓ Deploy & Bring Up/down to Optimize Resources & Cloud Spend

✓ Easily Customized as the UI or Business Needs Change

✓ Leverage Docker Containers, enabling Continuous Integration of Code Changes and Parallel Test Instances

✓ Reduce CapEx (Capital Expenditure) by Hosting in AWS (Amazon Web Services) Cloud

✓ Dynamically Support Infrastructure & Policy as Code, with Fugue…

✓ Configurations

✓ Configurations

✓Configurations

✓ Configurations

Target Service

DevTest

Typical DevTest SV Usage Scenario


C o n t i n u o u s - I n t e g r a t i o n v s . D e l i v e r y v s . D e p l o y m e n t

Continuous- Integration vs. Delivery vs. Deployment

Source ControlCommit Changes

BuildRun Build And Unit Tests

StagingDeploy to test Environment Run Integration

Tests, Load Test, & Other Tests

ProductionDeploy to Production Environment

V1.1

Automated Automated

Continuous Deployment

Continuous Delivery

Automatic Deploy

Approve Deploy

Continuous Integration


What tools/technologies should QE team get trained

Though Selenium continue to dominate, the following tools have been adopted faster

Web Driver IO Protractor Concourse

REPL is used to build web driver component

With DevOps, The need for Microservicestesting with spring server technologies will take higher precedence

Server 1 Server 2 Server 3

Service

By 2019, Spring Server will dominate the server market space (https://github.com/bartobri/spring-server)

GitHub will become the global platform and open source will be the future.

Brief list of various Test Automation tools that have been in faster adoption

https://concourse.ci/ - This is better than Jenkins - Shell script - runs on a FLOW.

https://gradle.org/Alternate to Maven

http://gulpjs.com/ - Supports Jasmine, Mocha - BDD tools -This is better than Cucumber

automate just about anything with a

minimum of effort

http://webdriver.io/ Selenium runner - reports in Allure - very powerful one for integration.

Webdriver CS

Visual Regression

Concourse as a server. Enables multiple deployment for seamless test execution

https://github.com/bartobri/spring-server

https://concourse.ci/

https://gradle.org/

http://gulpjs.com/

http://webdriver.io/


Operations Stabilization & Alignment with DevOps

DevOps Enabled Test Lab

1

2

3

4

Release Automation

Configuration Management

Environment Management


Automated DeploymentsStaging

QADev

Deployments Smoke Tests

Rapid Configuration Management Multiple Nodes across

technologies

Build Environments

Virtualized & Non - Virtualized

Staging

QADev

Health CheckInfrastructure

Application

Threshold evaluation

Virtual Server(s)

Physical server(s)

CI Server

Hosted Tools ManageMaintainMonitor


Continuous Automation & Delivery Pipeline

ORCESTRAED CONTINIOUS INTEGRATION PIPE LINE

Test

SCM Repo Adoption

Build Package Deploy Host / Stage

Jenkins

Requirements Document & verify

Container as a SCM service using Rancher/Docker

Environment Environment

Resources ResourcesPrivate Container Service

User Directory Container Registry

Ops ToolsDevOps Pipeline

Enterprise Micro service Stack

Reporting & Monitoring

On demand test bed using containers


Web & Mobile browsers

Advance Analytics

Dashboards Tableau

De

liver

y P

ipe

line

Co

nti

nu

ou

s A

uto

mat

ion

Requirements

Automated Testing

Test Automation Scripts

CI Enabled Execution

Results

Automated Performance DAST

Regression & Risk Based

Continuous Development

Coding

Compile & Build

Code Quality

SAST

Unit testing

Continuous Deployment

Deploy AppsRelease

Management

On Demand Infra

Collaborative Dashboard


Operations Enablement


Collaborative DevTest Platform & How does it help?

Build OperateIntegrate Deploy

Application Health Release Readiness Quality Metrics Go – No Go Decision Enabler

Release Progression Build Status Test Status Predict Analysis

Live

Da

shb

oa

rd


Test Management Tools

Performance Testing Tools

DAST ToolsMonitoring

Tools


Q&A

Quality Engineering in DevOps world a Strategic...

Documents

Transcript of Quality Engineering in DevOps world a Strategic...