Agile Testing - Build a Truly Agile Software Organization

Agile Testing BUILD A TRULY AGILE SOFTWARE ORGANIZATION

1/30/14

2 ©Alliance Global Services 2014

Today’s Presenter

As Vice President, Solution Engineering, Sreekanth has more than 14 years of senior technology leadership experience and leads Alliance’s Solution Engineering organization in developing cutting edge solutions. He is a respected technology thought leader with a focus in testing Agile projects, test process automation and metrics-based development approaches and utilizing lean techniques in software development.

Today’s Host

As Director of Marketing, Sharon Lee heads the marketing strategy and brand messaging for Alliance. With over 12 years of experience in both digital and traditional marketing, she is focused on the effective use of media for compelling brand messaging and creating successful marketing programs with measurable results that impact revenue. Sharon holds a B.A. from the University of Pennsylvania.


AGENDA


Agenda

• The case for QA in an Agile World

• Changing role of QA in an Agile World

• Role of QA in Agile organizations

• Profile of Testers and QA Analysts for these

organizations

• Automation best practices

• Metrics and measurements


The Case for QA in an Agile World


The Promise of Agile

Develop iteratively as business evolves

Deliver working software rapidly

Flexibility to change with business needs

Incorporate end-user feedback

Highly productive technical teams


Common pitfalls of Agile projects

Functionality/defect treadmill

High regression defect seepage

“Epic” failures

Non-functional requirements fail

Low effective team throughput


What Agile Assumes

Communication

Collaboration

Access to Stakeholders

Cross Functional Teams


Role of Testing in Agile Teams

Some Agile proponents did not see (or argued against) the need for QA specialists on an assembled team.

If you use TDD, Unit Testing and “Everyone Tests their own Stuff,” do we need a dedicated QA team?


Agile Layers of Testing

Prep

Development Start Wrapper Sprint 2 or 4 Week Sprints

JumpStart 24h

24h

Harden, Docs, Release

Vis

ion

Pro

du

ct B

ackl

og

Sprint Backlog Tested, Working

Software Increment

Product X

Sprint Review Sprint

Planning

Release Planning

Sprint 1 Sprint 2 Sprint n

3 - 5 Sprints

Prototyping Reqs Testing

Unit Testing Regression

Integration Compatibility Performance

Security

Quality Assurance

Acceptance Functional

Deployment


Role of QA in an Agile World


Organization empowers health plans to transform the performance of their provider networks, while significantly reducing medical, administrative and IT costs and simplifies the design, maintenance, reimbursement and performance management of provider networks, while facilitating provider-patient collaboration.

Development Objectives:

• Rapid release of new solutions and rich new features of their core Platform into market to take advantage of increased health care IT spending in the payer market due to health insurance reform (PPACA)

• Ensure high level of customer satisfaction dependent on user experience, maintainability, scalability and platform performance

• Ensure the highest levels of security and confidential data access meeting all regulatory needs and customer specific needs

Development Organization:

• TDD approach for Developers engaged in 2-week sprint cycles

• Enterprise development technologies

• Interfaces with several in-house and 3rd party development teams and applications

• SCRUM, Kanban and Lean techniques

Organization Profile


Agile Layers of Testing

Prep

Development Start Wrapper Sprint 2 or 4 Week Sprints

JumpStart 24h

24h

Harden, Docs, Release

Vis

ion

Pro

du

ct B

ackl

og

Sprint Backlog Tested, Working

Software Increment

Product X

Sprint Review Sprint

Planning

Release Planning

Sprint 1 Sprint 2 Sprint n

3 - 5 Sprints

Prototyping Reqs Testing

Unit Testing Regression

Integration Compatibility Performance

Security

Quality Assurance

Functional

Deployment

Acceptance


Testing Activities

Activity Developers Testers

Requirements Testing

Acceptance Unit Testing Functional/Exploratory Testing

Regression Testing

Compatibility Testing

Performance Testing

Security Testing

Deployment Testing


Quality Assurance

Activity Developers Testers

Test Strategy

Test Planning Process Compliance Root Cause Analysis

Metrics & Reporting


Automation in Agile


Automation Objectives


Layers of Automation Layered Automation Testing Strategy for end to end coverage, automated solutions for

each tier and testing on actual devices

UX Testing

Services & API Testing

Unit Testing


Automation - Best Practices

Automation is development

Automation Architecture

Lagging sprints

Non-functional

Integration with CI


Environments

Test Management

CI Integration

Reporting

Infrastructure


• Re-usable automated test suites with minimal maintenance effort

• Cross application E2E automated testing

• Excel based test suite

• Detailed test summary reports along with step wise description and screen shots

• Test scripts, test reports integrated with QMS

• Automated 55% of regression scenarios which includes 70% of business critical scenarios

Test Automation Framework


Leveraging:

An Agile test automation platform that drives quality and efficiency

Webdriver / Robotium QTP TestComplete Sikuli

Scriptless Testing

Automation Framework to develop rapidly

across the enterprise creating highly

maintainable suites

Automation Coverage for All

Popular UI and Service Interfaces

InsideEdge AFT utilizes a revolutionary plug-in

architecture that leverages multiple execution

engines to obtain access to UI and Web

Services

Collaboratively Develop Automation

Business Analysts, Functional Testers and Automation Scripters can

collaborate to develop automation as an integral part of testing – not

a separate and specialized activity


Metrics


Measuring Effectiveness Automation initiatives need metrics reporting and analysis as an integral part of the initiative to ensure success of the initiative.

• Metrics should capture both business level and execution level values to ensure efficiency is accurately captured and reported

• Metrics should not be left for the end of the sprint/month/project to capture and track

• Metrics measurements allow implementation level course corrections or expectations management easier

• Business Objectives outlined should be directly tied to the metrics

program and measured through out the initiative

• Assumptions and approximations can be made at the beginning of the initiative


Key Metrics for Agile Projects

• Use Validation points to evaluate application’s map with requirements

• Communicate to all stakeholders the coverage provided by formal testing

Coverage Metrics

• Number of defects identified in different stages of software lifecycle

• Number of defects created in different stages of software lifecycle

Defect Identification Efficiency

• Quality of software code

• Impact of technical or architectural choices Technical Debt

• Demonstrate the confidence of the testing team in the testing efforts executed on a release with the resources provided for testing Confidence Level


Leveraging:

Shortening time-to-market and enabling predictable delivery of software that works every time

Continuous Integration

Baked-In Quality: script-less automated

functional testing (xAFT), unit tests, continuous

integration (CI), automated builds, and

Clean Code analysis ensure ready to deploy

software without the wait

InsideView™ Toolbox

Growing collection of reusable tools and

components to solve common problems and

speed develop of robust web and mobile

applications

Application Lifecycle

Management (ALM)

Integrated Dashboard for managing user

stories (requirements), tasks, issues, backlogs,

burndowns, defects, quality & productivity

metrics give a one-stop view of all project

activity, progress, and results

Knowledge Management

Formal knowledge acquisition and evaluation

framework that focuses on knowledge

synthesis, assessment and retention so that

the full team understands users, features,

goals & processes and “gets it”


Automated Code Quality Analysis


Thank You Sreekanth Singaraju VP of Solution Engineering [email protected] www.allianceglobalservices.com

mailto:[email protected]


Coverage Metrics

Principle Communicate to all stakeholders the coverage provided by formal testing Dimensions • Use Validation points to evaluate application’s compliance with requirements • Weight them by business criticality Definition Designed Coverage = Designed Validation Points / All Validation Points (Estimated) Executed Coverage = Executed Validation Points / All Validation Points (Estimated) Best Practices • Keep coverage metrics simple • Introduce granularity to address business needs • Trend to show progress (or declines – great case for additional resources)


Defect Identification Efficiency

Principle Identify how defects are identified and created. Evaluate compliance of Agile principles. Dimensions • Number of defects identified in different stages of software lifecycle • Number of defects created in different stages of software lifecycle • Use key Software stages to draw valuable inferences

Definition Proportion of defects identified by each stage of a lifecycle – Ex – Testing – 56%, UAT – 18%, Production - 4%, other - 22% Best Practices • Start with defects identified first – then introduce defects created • Use “Other” to catch exceptions and introduce granularity incrementally • Factor in defect severity if that information is readily available


Technical Debt

Principle Provide a financial impact of technical choices on the overall product resulting from technology, process and implementation choices made during development Dimensions • Quality of software code • Unresolved defects that are business or quality critical • Impact of technical or architectural choices

Definition Cost estimates for redressing most critical (Business perspective) defects or technical choices Best Practices • Explosive- Handle and communicate with care • Be conservative – use a very strong criteria to evaluate “criticality” • Use several available tools to evaluate software code quality


Confidence Level

Principle Demonstrate the confidence of the testing team in the testing efforts executed on a release with the resources provided for testing. Informational • Test Execution Coverage • Expertise of resources to test the application • Quantitative results of defects including Defect identification efficiency

Definition Based on the evaluation of the 3 dimensions above rank each of them on a scale of 0(Poor) – 1(High). Overall confidence = (3.Coverage.Resources.Results)/(Coverage+Resources+Results)

Best Practices • Be prepared with supporting facts • Define clearly laid out criteria for ranking

Agile Testing - Build a Truly Agile Software Organization

Technology

Transcript of Agile Testing - Build a Truly Agile Software Organization