Clareice Chaney, Clyneice Chaney &

XBOSoft

Best Way to Contract an Outsourced Agile Test

Team Webinar

Best Way to Contract an Outsourced Agile Test Team

Clareice Chaney CPCM, PMPClyneice Chaney PMP, CQMgr.

XBOSoft info

• Founded in 2006• Dedicated to software quality

• Software QA consulting• Software testing services

• Offices in San Francisco, Beijing, Oslo and Amsterdam

On the XBO side:

Amy Jan

About the SpeakersClareice Chaney has over 30 years of experience in commercial and government contracting with an emphasis in information technology contracting and specific expertise in performance-based contracting techniques.

Clyneice Chaney brings over 20 years of testing, quality assurance, and process improvement experience. Clyneice holds certifications from the American Society for Quality as a Certified Quality Manager, QAI Global Institute's Certified Quality Analyst, and Project Management Institute's Professional Project Manager.

Housekeeping• Everyone except the speakers is muted

• Questions via the gotowebinar control on the right side of your screen

• Questions can be asked throughout the webinar, we’ll try to fit them in when appropriate

• General Q & A at the end of the webinar

• You will receive info on recording after the webinar

Next Webinar -July 30

• Mobile Testing Tutorial• JeanAnn Harrison• BYOD• Registration page in your

chat

Agenda

Outsourced Testing Concepts

Contracts & Testing

Procuring Agile Projects

Monitoring Agile Testing

Outsourced Testing: Typical Categories

Consulting Services Quality Assurance Quality Management

Staff augmentationFunctional testingIntegration TestingPerformance Testing Testability Assessment

Test managementTCOERisk –based testingTest AutomationAgile TestingTest Data Management

Managed quality servicesQuality center of excellenceOn-site advisory/ assessmentShared outcome QA

Outsourced Testing Challenges

• No single best testing practice

• Complexity of system under test

• Test automation

– Misaligned baselines– Operational

inefficiencies– Poor risk and

performance management

– Collaboration failures

Is There a Difference Outsourcing Agile?

Requirement

Design

Build

Test

Deploy ?

Contract

Traditional

Agile Foils controlling systemsDefines deliverable differently

Agile & Contracting: Understanding a Basic Contradiction

• You need an agreement to define a fair and professional relationship.

• Contracts typically don’t allow for quick or easy changes

• Poorly constructed contracts have the potential to derail business objectives

Impact of Agile on Contracts

• Co-location• Decision making authority• Customer participation• Pricing concerns• Cross-functional; not separate

teams– Test often before delivery and part

of development process– More automated test & analysis for

early buy identification

What’s the Fix?

• Modular contracting to provide flexibility. Several smaller acquisitions that:– Are easier to manage individually– Address complex IT objectives– Provide for delivery, implementation & testing of workable

systems solutions in increments– Allows subsequent increments to take advantage of changes in

technology or requirements

And…. Fix # 2

First phase is a typical service/

consulting contract

2nd phase is a series of fixed price contract where you have defined:• What to build • Time frame • How much it will

cost

Results: A framing contract that is really

about the collaboration and the

end goal. Defines: # of iterations and the T&Cs for both

phases

And Fix # 3: Focus On What You know

Contract Project Objectives rather than the solution

• Target Cost• Target Schedule• Business objectives with a high level

overview of how the SW will help• Time and resources budgets

Well defined planning and acceptance process for each iteration

Shared Benefit Contracts

• Goal is to provide financial incentives for meeting project objectives and penalties if they aren’t met

• Not technical objectives… business objectives

Road to Good Outsourced Test Performance

Supplier Performance

# 1 Identify outcomes

# 2 Determine outcome measures

# 3 Define Service Levels

# 4 Establish Monitoring Methods & Monitor Outcomes

Terminology

Work

OutputsBenchmarks Indicator Accomplishments

What you Need to Know?

Quality of Service Quality of Product

Defect Density

Quality Attributes

Status Quality Target

Number of Defects

Schedule

Test Effectiveness

Test Planning Effectiveness

Defect Detection

Adapted from James Bach, Rapid Software Testing.

Measuring the Suppliers OutcomesNumber of Tests Attempted by Given Time

Number of Tests Passed by a Given Time

Number of Bugs Found by Given Time

Test Effectiveness Ratio

Defect Detection Capability

Duration of Testing Processes

Defects by Category/ Severity

Product Defect Density

Backlog of Open Bugs

Product Quality Over Time

Defect Detection Capability

Defect Severity Index

Agile Testing Metrics

Copyright © 2011 RGCG, LLC 41

21

Customer Acceptance Tests Defect fixes & Story Changes

Burn Down Burn Up

http://xprogramming.com/xpmag/BigVisibleCharts

5P Performance Measurement Framework

Adapted from: Kalyana: Performance Measurement Framework for Outsourced Testing Projects (April 3, 2009)

Project: for each task or project monitor meeting it’s requirements, service levels

Process: monitor suppliers processes, particularly ones they agreed to perform

Product: monitor excellence of deliverables

People: monitor supplier staff- average experience; availability, capability

Price: $ amount saved, price variance

Setting Testing Service Quality Levels

• Timeliness of service• Accessibility and convenience • Accuracy• Staff Availability• Service Delivery: Courteous?• Adequacy of information

disseminated• Customer satisfaction• Condition and safety of facilities used

Summary

Questions & Answers

Follow us: @XBOSoftQuestions at: services@xbosoft.com

Thanks and hope to see you soon!