Introducing Test Automation into Agile Scrum SW Development

Shai Ziv, April 2013

w w w . n e s s . c o m© 2012 Ness Technologies

TopicsTopics

Introduction

Objectives

Terms & Overview

Key success factorsKey success factors

Methodology & Processes

Introduction Introduction –– The Uniqueness of VThe Uniqueness of V--NessNess

VV--Ness is acting as an expertise and excellence center Ness is acting as an expertise and excellence center

for Ness Global customers around the worldfor Ness Global customers around the world

500500 100100’s’s

500 500 Testing Experts Testing Experts

in Israelin Israel

Hundreds Hundreds

Successful ProjectsSuccessful Projects

4 4 Specializing Specializing

laboratories (India, laboratories (India,

East Europe, East Europe, Israel, Israel,

USA)USA)

Over Over 15 15 years of years of

experienceexperience70 70 active active customerscustomers

44 1515 7070

ObjectivesObjectives

Present the importance of Automation in Agile Scrum SW Development

Share the key success factors in this Automation project

Present the chosen Methodology and Processes

Terms & Overview Terms & Overview -- Agile Agile && Scrum TermsScrum Terms

Agile SW Development is a group of software

development methods based on iterative

(repeated cycles) and incremental

(smaller portions at a time) development.

http://en.wikipedia.org/wiki/Agile_software_development

Scrum is an iterative and incremental Agile software development framework and the

most popular form for Agile SW Development.most popular form for Agile SW Development.

► Roles (Product Owner, Development Team, Scrum Master, Stakeholders, Managers, QA)

► Sprints (A sprint is the basic unit of development in Scrum)

► Meetings (Daily Scrum, Backlog grooming, Scrum of Scrums, Sprint planning, Sprint Review,

Sprint retrospective)

http://en.wikipedia.org/wiki/Scrum_(development)

Terms & Overview Terms & Overview –– Project OverviewProject Overview

Description:

► Several C# – WPF – MVVM (Model View – View – Model) based applications,

used for Electrical Validation (EV) in vast number of labs around the globe.

• Business Logic complexity

• Thousands of scenarios

• Hundreds of HW components (Drivers, DDRs, CPUs, Instruments, Providers, etc.)

Goals:Goals:

► Test Automation Development

• Low maintenance

• High coverage

• Generic & Flexible

• Derivable

• Transparency

► Full Synergy with Agile Scrum SW Development

Terms & Overview Terms & Overview –– Initial StateInitial State

QA Team Structure► Team Lead + 5 Manual Testers

Efficiency► Manual Testing only

► Almost no synergy between QA and Development

► QA Cycle of 5 working days (1 Week per sprint atthe end of 3 Weeks development)

• Flood of bugs are report at the end of the sprint

► Low Testing Coverage (< 25%) 200

300

400

500

600

700

800

788 , 75.6%

252 , 24.2%

Bugs per Customer Type (2011)

Development

EV Teams

QA

► Low Testing Coverage (< 25%)

• More than 75% of the bugs were found by Customers

• More than 70% of the bugs detected were Critical or High

► Few environment configurations

► No ability of regression package derivations

► No ability of remote execution

► No Configuration Management for QA Artifacts

► Releases and Sprints are being delayed

Outcome► Reorganization

► Products were omitted

► Looking for a new concept

0

100

2002 , 0.2%

0

100

200

300

400

500

111 , 10.7%

164 , 15.7%

412 , 39.5%355 , 34.1%

Bugs per Severity (2011)

Low

Medium

High

Critical

Terms & Overview Terms & Overview –– Current StateCurrent State

QA Team Structure► 2 Test Automation Engineers

Efficiency► Full synergy between QA and Development

► Ongoing QA + QA Cycle of 1 day

at the end of the sprint

• Bugs are being fixed on a daily basis according

to prioritization criteria 100

200

300

400

500

21 , 4.0%29 , 5.6%

471 , 90.4%

Bugs per Customer Type (2012)

Development

EV Teams

QA

to prioritization criteria

► High Automation Testing Coverage (> 95%)

• More than 90% of the bugs were found by QA

while less than 6% of the bugs were found by Customers

• Less than 46% of the bugs detected were Critical or High

► Dozens of environment configurations

► Regression package derivations

► Ability of execution on customer sites

► Configuration Management for QA Artifacts

► Semi-automatic bug analysis

► Complete Transparency

0

10021 , 4.0%29 , 5.6%

0

20

40

60

80

100

120

140

160130 , 24.9%

152 , 29.2%147 , 28.2%

92 , 17.7%

Bugs per Severity (2012)

Low

Medium

High

Critical

Terms & Overview Terms & Overview –– Current StateCurrent State

Outcome► High Quality of the developed applications

► Releases and Sprints according to timeline

► Customer Satisfaction

► Recognition - Concept implementation in other teams

► Consulting for vast range of teams inside the organization

ROI► Implementation time of 6 months

► Extremity High ROI = 0.9

Key success factorsKey success factors

Planning

Full Synergy between QA and Development

Maximum match of Automation Tools

TransparencyTransparency

Methodology & Methodology & Processes Processes -- PlanningPlanning

How we’ve exactly achieved all that?

High Automation Testing Coverage

► Gap Analysis and Alignment• Documents

• Environments

• Development Milestones

• Automation Tools POC

Chosen:Chosen:

Server Side and Services Automation: Coded UI

UI Automation: Ranorex

► Planning • Test Documents (ENVs, Prioritization, ROI, Automation Efficiency, Test Suites & Packages)

• Test Automation Structure (Automatic Settings, Building Blocks, “Zero” Maintenance – Dynamic Tags, Binding)

• Bidirectional Requirements Tractability

• Defects Management

Methodology Methodology & & Processes Processes -- SynergySynergy

Full Synergy between QA and Development

► Meetings (Daily Sync, Design reviews, End of Sprint, etc.)

► Document Transparency (SharePoint)

► Traceability mechanisms (Requirements, Bugs, Change Lists, Diffs – Beyond Compare by Scooter SW)

Methodology & Methodology & Processes Processes –– Ongoing QAOngoing QA

Ongoing QA + QA Cycle of 1 day

Continuous Integration builds for ongoing daily QA feedback (TeamCity by JetBRAINS).

► Full responsibility of build process

► Early identification of bugs

► Changes are still “fresh” to the developers

► Almost no impact of other check-ins

► Extends application stability and readiness

► Releases and Sprints accordingto timelineto timeline

Methodology & Methodology & Processes Processes –– Test ExecutionTest Execution

►Test Execution Cycles

• Unit Tests execution as part of CI. Development Goal: “Make QA flabbergasted because everything just works!”

• Sanity Test Execution as part of CI.

• Functional Tests Derivations used by Developers andEV Teams.

• QA Test Cycles according to test packages• QA Test Cycles according to test packages

- Regression

- Full

- Long Run

• Error Handler and Recovery Mechanism for test execution completeness

Methodology & Processes Methodology & Processes –– Test ExecutionTest Execution

Dozens of environment configurations

► Parallel Regression testing on multiple ENVs using MTM - Lab Center by Microsoft (a Lab Management component that supports the creation of virtual machines and “clean” lab environments used for testing).

Methodology & Processes Methodology & Processes –– Test ExecutionTest Execution

Execution on customer sites

► Automatic Tests Execution on customer sites

• Using Automation Test packages Derivations during implementation and integration phases.

► Derivations by Developers and EV Teams (Internal Customers)

• Sanity tests before check-ins.

Methodology & Methodology & Processes Processes –– Configuration & Defects Configuration & Defects ManagementManagement

Configuration Management for QA Artifacts

► QA artifacts are saved in source control and are fully aligned with

the applications branches and versions (VSS)

Semi-automatic bug analysis

► Bug skeletons are automatically reported according to Product, Layer and Area (TFS by Microsoft)

► Tickets are being extended (Details, Step to reproduce, Priority, Assignment, etc.)

► Ticket are linked to Test Cases, Sprints and Requirements (MTM – Testing Center , TFS and ALM Rangers by Microsoft)

Methodology & Methodology & Processes Processes -- TransparencyTransparency

Complete Transparency

► Status &Reporting (MTM, SharePoint)

• Weekly and Monthly Reports

• Project Dashboards - provide near real-time display of development and test project information and status

- Unified Data Access Layer

Methodology & Processes Methodology & Processes -- TransparencyTransparency

Complete Transparency

► Status &Reporting

• Gamification

Thanks!Thanks!