ESTABLISHING REQUIREMENTS METRICS

1 / OCTOBER 2008 / EDS INTERNAL

ESTABLISHING REQUIREMENTS METRICSCampbell Ferenbach

Global Business Analysis Capability

Insert photo here

2 / OCTOBER 2008 / EDS INTERNAL BUSINESS ANALYSIS PERFORMANCE METRICS

Objective

This presentation outlines:

1) Why aren’t we all collecting metrics for Business Analysis?

2) What to count and how to count it.

3) Implementation - the challenges ahead.

…to share information and gain new ideas


Agenda

• Why do we need Metrics?

• Requirements for Metrics Strategy

• Approach

• Current State

• What to measure?

• Defining the BA Measurements

• Sample Reports

• Recap

• Questions?


Why do we need Metrics?

• Performance measures are needed to :

– Enable analysis of Business Analysis productivity and quality

• Drive productivity savings

– Determine effectiveness of, and justify investments in:• training, • process and • tool improvements

– Provide foundation for estimating

– Demonstrate professionalism to clients and BA industry

– Enable Competitive Benchmarking


Cost of Requirements Defects• “Deficient requirements are the single biggest cause

of software project failure. From studying several hundred organizations, Capers. Jones discovered that RE is deficient in more than 75 percent of all enterprises. In other words, getting requirements right might be the single most important and difficult part of a software project. Despite its importance, we know surprisingly little about the actual process of specifying software.” (Hubert F Hofmann, Franz Lehner, ”Requirements Engineering as a success factor in software projects”, IEEE Software, July-Aug, 2001)

• “Of the IT projects that are initiated, 5% to 15% will be abandoned before or shortly after delivery as hopelessly inadequate. Many others will arrive late and over budget or require massive reworking. Few IT projects, in other words, truly succeed… The most common reasons for software failures include:1. unrealistic or unarticulated project goals, 2. inaccurate estimates of need resources, 3. badly defined system requirements,…” (Robert N Charette, “Why Software Fails”, IEEE Spectrum, Sept 2005)

• “Analysts report that as many as 71% of software projects that fail do so because of poor requirements management, making it the single biggest reason for project failure” (Christopher Linquist, “required: Fixing the Requirements Mess,” CIO, Nov 15, 2005)

• “Us of experienced analysts versus inexperienced analysts can reduce project cost by 33%. Use of experienced and capable analysts can reduce project cost by 50%.” (Karl Wiegers, Software Requirements)

The above graph assumes the cost of fixing a defect in the phase in which it was inserted has a cost of $75.00

Defect Correction Cost

$-

$2,000

$4,000

$6,000

$8,000

$10,000

RequirementsDes ign

Code

System / In

tegration Testi

C lient A cceptance Te sti Produc tio n

Requirements Defect Correction Design Defect Correction

Code Defect Correction

• “A requirements defect that is left undetected until construction or maintenance will cost 50 to 200 times as much as it would have cost to fix at requirements time.” (Taming Wild Software Schedules, Steve McConnell)


Defining Performance Measures - CapabilityPerformance Metrics at the Project level are no longer adequate

– Projects are increasingly executed by “Virtual Teams”, with members typically geographically dispersed

– Some projects are staffed by “Centers of Expertise”, where staff specialize in specific technologies and functions in the delivery lifecycle

Capability Performance provides a finer level of detail than Projectmetrics

• Development has:SLOC / Staff Month OR Function Points / Staff Month

• Testing has:Test Cases executed / Staff MonthPre- & Post-Release defects

Performance Metrics need to be developed for the different typesof Capabilities– In order to improve the performance of a capability we need to understand

the current baseline for that capability– Measurements need to be developed to address both the quality of work

products created, as well as the quantity of the work products created by the capability


Requirements for BA Metrics Strategy

• Key Requirements:

–Enable measurement of Business Analysis performance

–Ensure measurements cover both the “quality” as well as the “quantity” of the work products

–Use data collected by standard EDS/HP tools

–Enable meaningful comparison across different project profiles

• Methodologies, target technologies, etc.

–Integrate with & complement existing metrics programmes


Approach

• Scope the Problem–Research existing productivity metrics internally & externally–Establish requirements for the Business Analysis productivity

baseline• Define Productivity Model

–Define the “Business Analysis” metrics –Define productivity model & data requirements

• Collect Baseline Data–Develop metrics collection tools and process –Collect initial data

• Analyse Data & Calibrate Model–Analyse data, calibrate model–Refine assumptions & model

• Document Baselines–Formally document baselines from which improvements will be

measured–Separate by project type, scale, methodology, more?


Industry Information

What does industry information suggest?– The industry in general measures Project and Product productivity and quality.

»Very little data is available on Business Analysis performance

»Yet, widespread recognition that poor quality requirements are the reason for many project failures

– The industry is starting to look at the quality criteria of requirement statements

»e.g.: analyzing the word and grammar usage for clarity & ambiguity

– Various tools are available to enhance the quality of requirement statements. Among them are:

»NASA’s Software Assurance Technology Center use a method of measuring the quality of a requirements document by looking at “Completeness, Consistency, Correctness, Modifiability, Ranking, Traceability, Non-ambiguity, and Verifiability” and normalizing these metrics against size (lines of text). (http://satc.gsfc.nasa.gov/support/SEW_DEC96/sel.html )

»Some requirements management tools highlight text which is questionable and propose alternative text. The list of questionable terms is maintained by the tool administrator. It does not provide an overall score of a document but is does help the user improve the quality of the document.

»Some tools identify certain types of incomplete requirements, such as missing alternatives following an ‘If’ statement.


Goal-Question-Metric (GQM)

• Decide on your goals • What do you want to fix/improve/optimize?

» Improve BA productivity!

» Improve quality of requirements!

• Ask your questions • What do you need to know to meet the goals?

» Did we get more or less productive after a change?

» Did we make more or fewer mistakes after a change?

• Identify your metrics • What do you need to measure to answer the questions

Let’s look at those…


What can we possibly count?

• What phenomena do we have to measure?– Requirements produced & Effort = requirements written per unit of time

– How to count? Not all requirements are equal…

– What to include in Effort?

• just BAs? Reviewers? Draft only? Rework? Other changes?

• We looked at our process and requirements…– Scope? No

• part of contracting, highly variable, BA portion not very quantifiable

– High-Level Requirements? Yes

• BA-specific work, produces approved, countable requirements

– Detailed Requirements? Yes

• BA-specific work, produces approved, countable requirements

– Tracking BA effort only? No

• Need to include reviewers and other contributors

• Don’t include client’s time


A Consistent, Quantitative Measure• A requirements management tool can tell us how many are in the

current project.

• But… Simply counting all requirements is too simplistic– Only want requirements produced for the current release/milestone – Also those that were changed or deleted in the current release.

• And… Not all requirements are equal – One Use Case involves much more effort than a single textual requirement

and we wouldn’t want to compare them as equals– And what about the requirements embedded in models such as class (or

data) models and business process models?

• So… Count text-based requirements and ‘measure’ the size and complexity of use cases– Either a lot of counting of lines in MSWord-based UCs or come up with an

intelligent counting tool– Need a streamlined, low-effort way of estimating key elements that would

give a reliable indication of the scale and complexity (later)


Quantity – What are we going to count?

• Number of High Level Requirements, includes:• Business Processes• High Level Requirements statements• Business Policies

• Number of Detailed Requirement Statements, includes:• Functional Requirements and Use Cases• Non-functional Requirements• Business Rules• Class Model

• Reworked and Changed Requirements, includes:• Number of requirements changed due to defects• Number of requirements changed due to external influences

• Function Points, includes:• Function Point Count when Detailed Requirements are approved• Function Point Count at end of project

• Effort, includes:• Effort to produce High Level and Detailed Requirements• Effort to Rework (due to defects) or Change (due to CRs) previously approved

requirements


Quality – What to Measure?

High Quality Requirements are:• Correct

» Requirements as stated are 100% correct• Complete

» Requirements as stated have no missing elements» All Requirements are captured» Requirements as stated have sufficient detail to be the foundation of the Design,

Coding and Test phases• Consistent

» Requirements as stated do not contradict other requirements• Unambiguous

» Requirements as stated are not open to interpretation• Verifiable

» Requirements as stated can be fully tested (through both the normal paths as well as the abnormal paths)

• Ranked for Business Priority» Requirements have Business Priority defined (e.g. numeric priority from 1 –

highest priority to 5 – lowest priority)• Strategic

» Requirements are aligned with Client Strategic Vision

We will measure the quality of the approved version of the requirements


Performance Metrics for BA Capability

To measure Business Analysis in terms of the Quantity and Quality Criteria, three BA Performance Metrics have been identified:

• Productivity (Quantity)– Answers “How many requirements were produced, for a given amount of effort?”

• Rework (Quality)– Answers “Did we produce high quality requirements?”

• Volatility (Stuff happens)– Percent of Requirements Statements and Use Cases modified, added or deleted

due to new client requests

– Answers “How much churn occurred after the requirements were approved?”


Defining Productivity

• The Outputs:–Number of Requirements Statement and Weighted Use Cases

• The Inputs:–Staff Months of effort

Productivity = Outputs / Inputs


DefectDefectChangeChangeRequestRequest

Analysis &Analysis &DocumentationDocumentation

Summary: Business Analysis Performance Metrics

ElicitationElicitation

Client Baseline signClient Baseline sign--offoff

ClientClient

ProductivityMetric

# Requirements# RequirementsPerPer

Staff MonthStaff Month

Require-ments

RequireRequire--mentsments

Amount of EffortAmount of Effort

Client forgot the Client forgot the CEOCEO’’s requirements s requirements and asked and asked ““Can you Can you

still add those?still add those?””

% Changed /Added % Changed /Added RequirementsRequirements

VolatilityMetric

ScopeScopeChangeChangeRequestRequest

Oops, thatOops, that’’ssnot right!not right!

% Reworked Reqts % Reworked Reqts due to defectsdue to defects

ReworkMetric

New BaselineNew Baseline

Corrected BaselineCorrected Baseline Approved BaselineApproved Baseline

Client Client Change Change RequestRequest


Performance Metrics Definitions

Productivity (2 Measurement Points)Initial Requirements Productivity:– # of Requirement Statements and Use Cases / Requirements Effort in Staff

Months– Requirements Effort includes all effort expended to achieve customer Approval of

the High Level and Detailed Requirements

Overall Project Productivity:– # of Requirement Statements and Use Cases / Business Analysis Effort in Staff

Months– Business Analysis Effort includes all:

»Requirements Effort, during High Level and Detailed Requirements phases»plus Rework Effort, during Delivery phases

Rework– % of Reworked Requirements / Total # of Requirements Statements + Use Cases

Volatility– % of Changed Requirements / Total # of Requirements Statements + Use Cases


Metrics TerminologyEach metric requires a clear definition with explicit “counting” rules to ensure the resulting data is comparable across teams and across time

The total Effort expended to modify the Requirements Statements and Use Cases, due to Change Requests associated with client requests for new/changed functionality.Volatility Effort

The total Effort expended to elicit, document, analyse and review the High Level and Detailed Requirements Statements and Use Cases UNTIL the customer’s formal acceptance of the Detailed Requirements. This include the efforts expended by all resources involved, excluding effort from resources associated with the client

Requirements Effort

The total Effort expended to modify the Requirements Statements and Use Cases, due to Change Requests associated with requirement defects.Rework Effort

The total Requirements Effort and Rework Effort to produce and maintain the approved Requirements Statements and Use Cases, over the duration of a given project lifecycle (e.g., Release)

Business Analysis Effort

The weighted count of all Detailed Use Cases (non-textual Requirements Statements)Use Cases

The count of all textual Detailed Requirement statements (non-Use Cases)Requirement Statements

The count of Requirement Statements and Use Cases added, changed, or deleted, due to requirement defects

Reworked Requirements

The count of Requirement Statements and Use Cases added, changed, or deleted, due to new client requests, AFTER Detailed Requirements Approval

The standard number of hours of effort available per individual per month

Definition

Changed Requirements

Staff Month

Unit of Measure


Issues & Lessons Learned

1. Metric Measurements only make sense in Managed Work environments

• Metrics are of little to no use in Staff Augmentation environments

• Without consistency in processes and methods, we can’t isolate what causes a change in a metric from one release to the next

2. Tools need to be defined and configured such that metrics are automatically produced with little to no additional effort

3. Metrics need to be generated by the standard toolset• So they can very easily be collected in the corporate metrics

repository

4. Very few Industry and EDS metrics exist at the BA Capability Level

• We just don’t have clear baselines

• We have to get started and accumulate our own


Sample Reports…

• Project Metrics Report

• Project Summary Report






ClientClient

Require-ments


ProductivityMetric







VolatilityMetric




ReworkMetric




Summary: Business Analysis Performance Metrics


Recap

This presentation set out to cover:

1) Why aren’t we all collecting metrics for Business Analysis?

2) What to count and how to count it.

3) Implementation - the challenges ahead.

…to share information and gain new ideas






ClientClient

Require-ments


ProductivityMetric







VolatilityMetric




ReworkMetric




Business Analysis Performance Metrics

25 / OCTOBER 2008 / EDS INTERNAL

EDS5400 Legacy DrivePlano, TX 75024+1 972 605 [email protected] or eds.com

EDS and the EDS logo are registered trademarks of Hewlett-Packard Development Company, LP. HP is an equal opportunity employer and values the diversity of its people. ©2008 Hewlett-Packard Development Company, LP.

ESTABLISHING REQUIREMENTS METRICS

Documents

Transcript of ESTABLISHING REQUIREMENTS METRICS