Is the QA Department Dead in an Agile World?

IS THE QA DEPARTMENT DEAD

IN AN AGILE WORLD

Delivering better software quality on Agile projects

9/26/13

2 ©Alliance Global Services 2013

Today’s Presenter

As Vice President, Solution Engineering, Sreekanth has more

than 12 years of senior technology leadership experience and

leads Alliance’s Solution Engineering organization in developing

cutting edge solutions. He is a respected technology thought

leader with a focus in testing Agile projects, test process

automation and metrics-based development approaches and

utilizing lean techniques in software development.

Today’s Host

As Director of Marketing, Sharon Lee heads the marketing

strategy and brand messaging for Alliance. With over

12 years of experience in both digital and traditional marketing,

she is focused on the effective use of media for compelling brand

messaging and creating successful marketing programs with

measurable results that impact revenue. Sharon holds a B.A.

from the University of Pennsylvania.


AGENDA


Agenda

• The case for QA in an Agile World

• Changing role of QA in an Agile World

• Role of QA in Agile organizations

• Profile of Testers and QA Analysts for these

organizations

• Automation best practices

• Metrics and measurements


The Case for QA in an Agile World


The Promise of Agile

Develop iteratively as business evolves

Deliver working software rapidly

Flexibility to change with business needs

Incorporate end-user feedback

Highly productive technical teams


Agile Challenges

Functionality/defect treadmill

High regression defect seepage

“Epic” failures

Non-functional requirements fail

Low effective team throughput


What Agile Assumes

Communication

Collaboration

Access to Stakeholders

Cross Functional Teams


Is QA Dead in this Agile World?

Some Agile proponents did not see (or argued against)

the need for QA specialists on an assembled team.

If you use TDD, Unit Testing & “Everyone Tests their

own Stuff” do we need a dedicated QA team?


Agile Layers of Testing

Prep

Development Start Wrapper Sprint 2 or 4 Week Sprints

JumpStart 24h

24h

Harden, Docs, Release

Vis

ion

Pro

du

ct B

ackl

og

Sprint Backlog Tested, Working

Software Increment

Product X

Sprint Review Sprint

Planning

Release Planning

Sprint 1 Sprint 2 Sprint n

3 - 5 Sprints

Prototyping Reqs Testing

Unit Testing Regression

Integration Compatibility Performance

Security

Quality Assurance

Acceptance Functional

Deployment


Specialization Has Benefits


Role of QA in an Agile World


Organization empowers Health plans to transform the performance of their provider networks, while significantly reducing medical, administrative and IT costs and simplifies the design, maintenance, reimbursement and performance management of provider networks, while facilitating provider-patient collaboration.

Development Objectives:

• Rapid release of new solutions and rich new features of their core Platform into market to take advantage of increased health care IT spending in the payer market due to health insurance reform (PPACA)

• Ensure high level of customer satisfaction dependent on user experience, maintainability, scalability and platform performance

• Ensure the highest levels of security and confidential data access meeting all regulatory needs and customer specific needs

Development Organization:

• TDD approach for Developers engaged in 2-week sprint cycles

• Enterprise development technologies

• Interfaces with several in-house and 3rd party development teams and applications

• SCRUM, Kanban and Lean techniques

Organization Profile


Agile Layers of Testing

Prep

Development Start Wrapper Sprint 2 or 4 Week Sprints

JumpStart 24h

24h

Harden, Docs, Release

Vis

ion

Pro

du

ct B

ackl

og

Sprint Backlog Tested, Working

Software Increment

Product X

Sprint Review Sprint

Planning

Release Planning

Sprint 1 Sprint 2 Sprint n

3 - 5 Sprints

Prototyping Reqs Testing

Unit Testing Regression

Integration Compatibility Performance

Security

Quality Assurance

Functional

Deployment

Acceptance


Testing Activities

Activity Developers Testers

Requirements Testing

Acceptance

Unit Testing

Functional/Exploratory Testing

Regression Testing

Compatibility Testing

Performance Testing

Security Testing

Deployment Testing


Quality Assurance

Activity Developers Testers

Test Strategy

Test Planning

Process Compliance

Root Cause Analysis

Metrics & Reporting


Profile of Testers & Best Practices


Technology Transformation - 2002


Distributed Teams


Communication Realities


Skillsets for an Agile Tester

Risk Assessment

Test Strategy & Planning

Test Design & Execution

Requirements & Acceptance

Metrics

Data

Deployment

CI

Cloud

UX


Automation Best Practices


Automation Objectives


Layers of Automation Layered Automation Testing Strategy for end to end coverage, automated solutions for

each tier and testing on actual devices

UX Testing

Services & API Testing

Unit Testing


Automation - Best Practices

Automation is development

Automation Architecture

Lagging sprints

Non-functional

Integration with CI


Environments

Test Management

CI Integration

Reporting

Infrastructure


• Re-usable automated test suites with minimal maintenance effort

• Cross application E2E automated testing

• Excel based test suite

• Detailed test summary reports along with step wise description and screen shots

• Test scripts, test reports integrated with QMS

• Automated 55% of regression scenarios which includes 70% of business critical scenarios.

Test Automation Framework


Metrics


Measuring Effectiveness

Automation initiatives need metrics reporting and analysis as an integral part of the

initiative to ensure success of the initiative.

• Metrics should capture both business level and execution level values

to ensure efficiency is accurately captured and reported

• Metrics should not be left for the end of the project to implement

• Metrics measurements allow implementation level course corrections or

expectations management easier

• Business Objectives outlined should be directly tied to the metrics

program and measured through out the initiative

• Assumptions and approximations can be made at the beginning of the

initiative


Key Metrics for Agile Projects

• Use Validation points to evaluate application’s map with requirements

• Communicate to all stakeholders the coverage provided by formal testing

Coverage Metrics

• Number of defects identified in different stages of software lifecycle

• Number of defects created in different stages of software lifecycle

Defect Identification Efficiency

• Quality of software code

• Impact of technical or architectural choices Technical Debt

• Demonstrate the confidence of the testing team in the testing efforts executed on a release with the resources provided for testing Confidence Level


Metrics & Effectiveness Measurements Detailed execution metrics will be captured to glean insights about test process, execution and

SDLC process.


Thank You Sreekanth Singaraju

VP of Solution Engineering

[email protected]

www.allianceglobalservices.com

mailto:[email protected]


Coverage Metrics

Principle

Communicate to all stakeholders the coverage provided by formal testing

Dimensions

• Use Validation points to evaluate application’s compliance with requirements

• Weight them by business criticality

Definition

Designed Coverage = Designed Validation Points / All Validation Points

(Estimated)

Executed Coverage = Executed Validation Points / All Validation Points

(Estimated)

Best Practices

• Keep coverage metrics simple

• Introduce granularity to address business needs

• Trend to show progress (or declines – great case for additional resources)


Defect Identification Efficiency

Principle

Identify how defects are identified and created. Evaluate compliance of Agile

principles.

Dimensions

• Number of defects identified in different stages of software lifecycle

• Number of defects created in different stages of software lifecycle

• Use key Software stages to draw valuable inferences

Definition

Proportion of defects identified by each stage of a lifecycle – Ex – Testing –

56%, UAT – 18%, Production - 4%, other - 22%

Best Practices

• Start with defects identified first – then introduce defects created

• Use “Other” to catch exceptions and introduce granularity incrementally

• Factor in defect severity if that information is readily available


Technical Debt

Principle

Provide a financial impact of technical choices on the overall product resulting

from technology, process and implementation choices made during

development

Dimensions

• Quality of software code

• Unresolved defects that are business or quality critical

• Impact of technical or architectural choices

Definition

Cost estimates for redressing most critical (Business perspective) defects or

technical choices

Best Practices

• Explosive- Handle and communicate with care

• Be conservative – use a very strong criteria to evaluate “criticality” • Use several available tools to evaluate software code quality


Confidence Level

Principle

Demonstrate the confidence of the testing team in the testing efforts executed

on a release with the resources provided for testing.

Informational

• Test Execution Coverage

• Expertise of resources to test the application

• Quantitative results of defects including Defect identification efficiency

Definition

Based on the evaluation of the 3 dimensions above rank each of them on a

scale of 0(Poor) – 1(High). Overall confidence =

(3.Coverage.Resources.Results)/(Coverage+Resources+Results)

Best Practices

• Be prepared with supporting facts

• Define clearly laid out criteria for ranking

Is the QA Department Dead in an Agile World?

Technology

Transcript of Is the QA Department Dead in an Agile World?