Ambiguity_Reviews_Improving_Bus_Reqs_Schanta_StarEast2014

| © 2014, Cognizant 0 © 2014, Cognizant | All rights reserved. The information contained herein is subject to change without notice.

Susan Schanta

Process, Quality & Consulting

April, 2014

Ambiguity Reviews

Building Quality Requirements

| © 2014, Cognizant 1

Agenda

The Cost of Quality 2

What is an Ambiguity Review? 3

Ambiguity Review Place in the SDLC 11

Ambiguity Review Classification 14

Ambiguity Review Types 20

Ambiguity Review Template 23

Ambiguity Review – Keyword Searches 27

Ambiguity Review – Content Review 30

Ambiguity Review Metrics 35

Ambiguity Review – Benefit to QA 40

Ambiguity Review – Be Nice About It! 42

Appendix 44


The Cost of Quality

2

• IRS Tax Systems modernization project spent $3.3b before

canceling (Federal Computer Week March, 2002)

• Time Warner Communications spent $1b on a failed

information system to break into the Residential Telephone

Business (Computerworld May, 1997)

• Sainsbury, a British food retailer wrote off $526m

invested in an Automated Supply-Chain Management

System.

• Ford Motor Company spent $400m on a Purchasing

System before abandoning it in 2004

50% of defects are due to requirements problems (Schwaber, 2006)


What is an Ambiguity Review?


What is an Ambiguity?

Ambiguity can be defined as…

1. Use of words that allow alternative interpretations

2. An unclear, indefinite, or equivocal word, expression,

meaning, etc.

3. The possibility of interpreting an expression in two or more

distinct ways

4. Doubtfulness or uncertainty of meaning or intention: to speak

with ambiguity; an ambiguity of manner. (dictionary.com)


What is an Ambiguity Review?

An Ambiguity Review is a formal review process that focuses

on the identification of ambiguities in the language, structure

and logic of a requirement.

Provides a measure of whether requirements are

quantitative, clear, correct and complete.

Eliminates requirements defects from being identified in

later phases by building quality into the product. (Bender)


What Are the Benefits of Ambiguity Reviews?

Ambiguity Benefits realized…

• Scope Creep is controlled

• Reduced cost of maintenance

• Reduced number of change requests

• Increased traceability from requirements to test cases

Defects per requirement

Tangible Results…

Healthcare Insurer

• Reduced Defect Leakage from 35% to 8%

• Introduced BA Style Guide to drive down ambiguities

Fortune 50 Insurance Company #1

• Reduced ambiguities from an avg. 7 per Req. to 2 per Req.

• Introduced BA Style Guide to drive down ambiguities

Fortune 50 Insurance Company #2

• 63% of all defects were traced to Requirement Defects


Business Users Focus on Happy Path

Impact to business users

• Who focus on Happy Path

Results identify…

• Open ended questions

What are your pain points?

• Requirements lack input/output, alternative flows, error

conditions and constraints

• Development makes assumptions to fills in gaps in the

requirements

• Potential Defect leakage to production

Why do an Ambiguity Review?


Development Drives Requirements

Impact to Development

• Who drive application behavior based on assumptions

where requirement gaps exist

Results identify…

• Higher maintenance is a higher percentage of IT budget

Limits dollars spent on revenue producing software

Increases redo work

• Change requests trend higher

• Quality Assurance is expected to test quality in



QA Builds In Quality, Right?

Impact to QA

• Who are expected to test quality into the product

Quality cannot be tested into the product…

Results identify…

• Test Cases may not address..

Gaps in user scenarios

Gaps in alternative flows (error conditions)

Gaps in business rules

• Confusion and delays with requirements often results in a

compressed schedule for testing

Increases redo work

• Defect leakage into production



Ambiguity Review Benefits

• Limit Scope Creep

• Increase Requirements Traceability to Test Cases

• Reduce Defect Leakage to Production

• Improves Estimation Accuracy

• Reduce Cost of Maintenance

− How much is your organization spending to maintain systems today?

• Reduces redo work

• Increases velocity of Test Case creation

• Increase BA productivity and work product

− Elicitation Checklists

− BA Style Guide

How do I measure success?


Ambiguity Review Place in the SDLC


Ambiguity Reviews in Requirements Phase

Requirements Phase

• Elicitation Preparation

• Elicitation Session

• Document Elicitation Results

• Confirm Elicitation Results

BA Peer Review

Ambiguity Review

Final Review & Signoff by Key Stakeholders

Note: Ambiguity Reviews are SDLC agnostic. The process

can be applied to any lifecycle and any document format.

Building Quality In…


Ambiguity Review in the Requirements Phase Where do Ambiguity Reviews fit into the Lifecycle?


Ambiguity Review Classification


Ambiguity of Reference & Ambiguous Statements

A condition when a requirement uses words such as pronouns,

adjectives, adverbs and verbs that can be interpreted differently

based on the reader’s view.

The report shall run frequently.

Ambiguity: What is the name of the report?

Ambiguity: What are the data elements in the report?

Ambiguity: Frequently is not measurable

Ambiguity: What is the report generation schedule?

Ambiguity: What happens if there is no data for the report?

Ambiguity: What happens if the report fails?

(Wiegers, Bender)


Boundary Ambiguity

A condition when the author uses terms - among or up to. The

scope of the requirement is ambiguous because the stated

requirement can be interpreted in multiple ways.

Example:

1. If an employee makes less than $20,000 per year, the

employer pays 100% of the healthcare premium.

2. If an employee makes more than $20,000 per year, the

employer pays 50% of the healthcare premium for the

employee.

Ambiguity: What if an employee makes exactly $20,000?

(Wiegers, Bender)


Built-In Assumptions

A condition when the author assumes that all consumers of the

document will have the same level of domain knowledge

• Industry domain knowledge

• Subject domain knowledge

• Functional knowledge

• Environmental knowledge

Example: The system must apply the same limitations to

searches for existing groups as currently exists in Google Search.

Ambiguity: The requirement assumes the reader knows how the

functionality exists today.

(Wiegers, Bender)


Dangling Else

A condition when a requirement states expected results (what

normally happens) but does not state exceptions and error conditions.

Dangling Else Can Shall

Could Should

Is one of Will

Must Would

Example: The employee address type shall be either house,

apartment or condominium.

Ambiguity: The requirement does not consider exception conditions

such as PO Box.

(Wiegers, Bender)


Etc.

Etcetera is not a quantifiable measurement that can be confirmed so

it is considered totally ambiguous. (Phrases or sentences ending with

etc…)

Example: Subscribers shall identify themselves with unique

information (policy number, social security, etc.) when they call

Customer Care for information about their policy.

(Wiegers, Bender)


Ambiguity Review Types


Categorizing Ambiguities to Support Metrics

Ambiguity Categorizations will help with…

• Tracking patterns of ambiguities

− For a Business Analyst

− For the Program

• Building Elicitation Checklists

− Lessons learned turned into questionnaires

− Mentoring sessions for the BA Team

• Team Performance Measurements

− Scope Management

− Defect Leakage to Production

− Requirement Defects

Purpose of Ambiguity Types


Ambiguity Type Description

Ambiguous Term Terms (Phrase or Word) used in requirements which can be interpreted

by the reader in multiple ways e.g. frequently, occasionally, efficiently

Conflicting Requirement

Requirements which contradict each other – either in the same

document or across multiple documents.

e.g. The field name is Effective Date but the data type is defined as an

integer

Glossary Word or acronym used in requirements that is new or not commonly

used but has not been defined in the Glossary/Definitions section.

Grammar, Spelling

& Wording

Grammar, spelling corrections and proposed wording improvements to

increase clarity of the requirement

Incomplete Requirement

Incomplete requirement or statement describing conditions when

information is not fully detailed preventing design or test validation

e.g. The system shall handle 15-25% increase in the second year

Missing Requirement

Missing requirements that were not documented or may not have been

elicited from the business user.

e.g. Missing requirements – alternative flows, business rules,

exceptions and error conditions (Questions – What, When, Where)

Unclear Requirement

Requirements or statements requiring further clarification to allow the

reader user to fully understand the requirement (Questions – How,

Why, What do you mean by…)

(Wiegers, Bender)


Ambiguity Review Template


Ambiguity Review Template

• Transfer requirements document to Excel template

− Revision History

− Introductory information (free text)

− Business Requirements

− Data presented in tables such as Glossary, Field Elements

and Financial Information segregated to its own worksheet

• Ambiguity Columns to the Right

− Ambiguity Type

− Ambiguity Description

Transform the Requirements Document to Excel


Requirements Worksheet Example Not all columns need to be transferred to the Ambiguity Template…


Glossary Worksheet Example Text transferred to Excel template…


Ambiguity Review – Keyword Searches


Ambiguity Review Keywords

An initial review for keywords helps to identify Incomplete

Requirements

All files are transmitted daily.

All 820 Payments Files shall be transmitted to the Federal and

Maryland Jurisdiction after the nightly batch jobs complete

• The Federal 820 Payment File shall be sent daily during the

transmission window between 11 PM and 4 AM EST

• The Maryland 820 Payment File shall be sent daily during the

transmission window between Midnight and 4 AM EST

• If no payments are made for a jurisdiction, an empty file shall

be sent to the jurisdiction

Clues to Finding Ambiguities…


Ambiguity of

Reference Ambiguous

Adjectives Ambiguous Adverbs Ambiguous Variables

Ambiguous

Verbs

above ordinary infrequently the database derive

below rare intuitively the field determine

it same just about the file edit

such seamless more often than not the frame enable

the previous several more or less the information improve

them similar mostly the message indicate

these some nearly the module manipulate

they standard normally the page match

this the complete not quite the rule maximize

those the entire often the screen may Ambiguous Adjectives transparent on the odd occasion the status might

all typical ordinarily the system minimize

any usual rarely the table modify

appropriate valid roughly the value optimize

custom Ambiguous Adverbs seamlessly the window perform

efficient accordingly seldom Ambiguous Verbs process

every almost similarly adjust produce

few approximately sometime alter provide

frequent by and large somewhat amend support

improved commonly transparently calculate update

infrequent customarily typically change validate

intuitive efficiently usually compare verify

invalid frequently Ambiguous Variables compute

many generally the application convert

most hardly ever the component create

normal in general the data customize

(Bender/Wiegers)


Ambiguity Review – Content Review


Ambiguity Review for Content

• Ambiguities in content focus on the following

− Conflicting Requirements

− Incomplete Requirements

− Missing Requirements

− Unclear Requirements

• Questions of Who, What, When, Where & Why

• Questions of How tend toward design questions

− Only use to clarify expected behavior or outcome

− Do not use to ask how the system will process the function

behind the scenes

Clues to Finding Ambiguities…


Sample Ambiguity Questions

• What is the name of the file?

• Who is the user?

• What permissions does the user need to review the file?

• When is the file sent?

• What are the contents of the file?

• Where is the file sent?

• What are the business rules to validate the file?

• What are the error messages displayed when a business rule

validation fails?

• What happens if the file is corrupted and can’t be read?

• What happens if the file fails to be generated?

• Is an alert sent if a file is late, corrupt or fails?

• Who is the alert sent to?

Who, What, When, Where, Why…


Ambiguity Review Example

If the policy is a subscriber + spouse/domestic partner +

dependents policy, subscriber wants to cancel from the policy,

Exchange will send us 834 file to enroll the spouse/ domestic

partner as the subscriber under a new SID and the dependents

will remain as dependents under the new subscriber.

Note: Subscriber + Family

Content Questions Raised for this Requirement


Ambiguity Questions Posed…


Ambiguity Review Metrics


Ambiguity Review Metrics Patterns within the Requirements Document

3

6

Top 10 Ambiguities by Type Total

Ambiguous Term 55

Grammar Spelling & Wording 79

Incomplete Requirements 59

Unclear Requirements 57

Total Ambiguities 250


Program Level Ambiguity Review Metrics Patterns across the Requirements Document


Program Level Ambiguity Review Metrics

Patterns across the Requirements Document


Program Level Ambiguity Review Metrics

Patterns across the Requirements Document

Ambiguity Review Summary Totals

Total Requirement Considered* 3,737

Total Ambiguous Observations 26,478

Top 10 Ambiguities 7,463

Percent Ambiguities in Top Ten 28%

Average Ambiguities Per Requirement 7*Requirements include Non-Requirement Sections such as Introduction, Purpose, etc.


Ambiguity Review – Benefit to QA


QA Benefits from Ambiguity Review

• Geographically dispersed teams benefit from the additional

level of detail documented in the requirements

• Test Case Creation is simplified by clear requirements

− Normal Flow & Alternative Flows detailed in Use Cases

− Error conditions with error messages

− Business rules and constraints

− Data requirements

− Expected Outcome

• Traceability to requirements provides QA Engineer with path

to use cases and functional/non functional requirements

− Diagnosis of defect types clear

• Requirements vs. Code vs. Test Case

Quantifiable Impact


Ambiguity Review – Be Nice About It!


Business Analysts Have Feelings

Ambiguity Reviews are meant to be objective and focus on the

requirements content. However… You are critiquing a

Business Analyst’s work.

• Recognize the Business Analyst’s work effort

• Recognize the Business Analyst’s challenges with eliciting

requirements from business users.

• Be courteous when providing feedback.

Write Ambiguity Questions that are quantitative, clear, correct

and complete!

Be respectful of your feedback


Appendix


Requirements Review Phase


References

Writing High Quality Requirements

By Karl E. Wiegers, 2006

The Ambiguity Review Process

By Richard Bender

Assessing the Impact of Poor Requirements on Companies

By Keith Ellis

Executive Guide to Evaluating Business Requirements Quality

By Keith Ellis

Getting Consensus on Business Requirements – Tips and Traps

By Keith Ellis

The Quest for Good Requirements

By Dr. Martin Schedlbauer, 2011

How to Prevent the Negative Impacts of Poor Requirements

By Sergey Korban, April 30, 2013

The Business Value of Better Requirements

By Karl E. Wiegers, 2006

Ambiguity_Reviews_Improving_Bus_Reqs_Schanta_StarEast2014

Documents

Transcript of Ambiguity_Reviews_Improving_Bus_Reqs_Schanta_StarEast2014