Copyright QSM Associates, Inc. 1 Michael C. Mah Managing Partner QSM Associates, Inc. 75 South...

1Copyright QSM Associates, Inc.

Michael C. MahManaging Partner

QSM Associates, Inc.75 South Church Street

Pittsfield, MA 01201413-499-0988

Fax 413-447-7322e-mail: [email protected]

Presentation for Chicago SPIN

January, 2002

Deadline-Driven Software Project EstimationNegotiating Trade-offs and Risks

Web Site: www.qsma.com


Background:QSM Associates

“The Metrics Company”

QSM Software Lifecycle Management Tools (SLIM Suite) Used Worldwide by Fortune 100 Clients to Measure, Estimate, and Control Software Development

“Management By The Numbers” - “Numbers into Pictures”


Partial List of QSM Clients

British Telecom EDS

Rockwell Intel BellSouth IBM Global Services

Sprint

Honeywell Computer Sciences Corp

United Technologies

GTE

Compaq

Keane

Boeing

Royal Bank of Canada

Lockheed Martin

Alcatel


Worldwide Software Productivity Trend

The Good News: Over the last decade, sw dev’t productivity has increased fairly consistently. 10% faster speed, 25% less cost, about every 2.5 years.

The Bad News: It’s Not Enough. Demand continues to outstrip capacity.

Companies are reporting growing backlogs ranging from 7 months to over 2 years.

The Badder News: Continued labor shortages. The More Badder News: Don’t expect any

schedule relief in an “Internet Speed” economy.


Industry Overrun Statistics

$250 Billion + Spent on IT Application Development

31% of Projects Will Be Cancelled, Representing $81 Billion in Losses

52.7% of Projects Will Overrun by >189%

Only 16.2% On-time,Under Budget

But with only 42% Original Functionality!

* Source: Standish Group, Dennis MA,


Anyone Know What This Is?


Reality Bites

“We don’t have the luxury of determining our schedules. They’re told to us. Then, given the time frame, we try to tell the client what we can build. In the end, we usually wind up working lots of overtime, because they want everything.”

- Manager of Development Wall St. Financial Firm


Size

Probabilityon

Time

Scenario: Size Growth,or “Feature Creep”

50%

75%

25%

A B C


Good to Know...

"As impressive as growth of the software industry has been, it is outpaced by growth of software-related litigation. It is not unusual for a large software development organization today to have upwards of 50 active cases on its hands."

Tom DeMarco, Cutter IT Journal


“Most litigation ends up focused on [lack of] measurement, management, requirements practice, or some combination thereof.”

“Organizations that can’t or don’t measure themselves in a fairly systematic way are at a huge disadvantage in litigation. If you are deficient at measurement and the other side is on top of it, then the jig is up for you.”

Tim ListerCutter IT Journal

Good to Know...


QSM Mixed Application Data Base

Effective Source Lines of Code

Calandar

Months

0.1

1

10

100

1000

100 1000 10000 100000 1000000 10000000

Real Time

Engineering

Info Systems

Impossible Zone

Are Deadlines/Plans in the “Impossible Zone”?


Building Your Own Benchmarks

Main Build Time vs. Size

New and Modified Size10 100 1000 10000

Months

0.1

1

10

100 Main Build Effort vs. Size


Person-M

onths

0.1

1

10

100

1000

10000

MTTD - 60 Days of Production vs Size

New and Modified Size1 10 100 1000 10000

MT

TD

- 60 Days

0.01

0.1

1

10

100

1000 PI Average and Stand. Dev.

PI

1 3 5 7 9 11 13 15 17 1921 23 25 27 29 31 33 35 37 39

Num

ber of Projects

0

2

4

6

8

All Projects in Sample Avg. 1 Sigma


Main Build Time vs. Size


Months

0.1

1

10

1002001 BASELINE SCHEDULE TREND

Main Build Effort vs. Size


Person-M

onths

0.1

1

10

100

1000

100002001 BASELINE EFFORT TREND

MTTD - 60 Days of Production vs Size

New and Modified Size1 10 100 1000 10000

MT

TD

- 60 Days

0.01

0.1

1

10

100

10002001 BASELINE RELIABILITY TREND

PI Average and Stand. Dev.

PI

1 3 5 7 9 11 13 15 17 1921 23 25 27 29 31 33 35 37 39

Num

ber of Projects

0

2

4

6

8 2001 BASELINE PI

All Projects in Sample Avg. 1 Sigma

Building Your Own Benchmarks


Ed Yourdon on “Sizing”..

“Studies by the Carnegie Mellon SEI indicate that the most common failing ofLevel 1 (Ad-hoc) software organizations is an inability to make size estimates accurately.”


Ed Yourdon on “Sizing”..

“If you underestimate the size of your next project, common sense says that it doesn’t matter which methodology you use, what tools you buy, or even what programmers you assign to the job.”


Many Functional Metrics Can be used to Represent S/W Size

Number of subsystems

Number of entities

Number of function points

Number of modules

Number of objects

Number of programs

Number of SLOC

Number of object instructions


Breaking things Down to Size

Existing Code,Database 4GL, Forms, PL/SQL,Pro*C, Reports,Utilities, etc.(Item C)

New Code(Item A)

Modified Code(Item B)

Existing,or “Base Code”


When Time & Effort are “Fixed” Something’s Got to Give...

SizeTime

Effort

Defects


“Feature Creep” - Impact on Quality with Fixed Schedule

Time Profile

11

12

13

14

15

16

17

18

Mo

nth

s

1 2 3 4 5Solutions

FOC MTTD Profile

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Day

s

1 2 3 4 5Solutions

Size Profile

60

70

80

90

100

110

ES

LOC

(thou

san

ds)

1 2 3 4 5Solutions

Solution 4

TimeEffortUinf CstPk StaffMTTDSize

14.06116.40

106712.922.15

87000

MonthsPM$ 1000PeopleDaysESLOC

100% Prob

38% Prob100% Prob 1% Prob

PI 17.0

Schedule Fixed at 14 Months

Size Increases in 5KSLOCIncrements from 72K to 92K

Quality Levels Cut in Half - MTTD 3.5 Days to 1.75 Days


Overall Project Risk:Green (Minimal Risk)

Schedule, Cost, andQuality Targets all at 80% Probability orBetter

Staffing Profile

0

5

10

15

200 1 2 3 4 5 6 7 8

Sta

ff

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 *Jan'97

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'98

Feb Mar Apr May

Feas

FD

MB

Maint

0 = FSR

1 = PDR

2 = CDR

3 = FCC

4 = SIT

5 = UOST

6 = IOC

7 = FOC

8 = 99R

RISKTimeEffortUinf CstMin Pk StaffMax Pk StaffFOC MTTD

% 0 10 20 30 40 50 60 70 80 90 100

TimeEffortUinf CstPk StaffMTTDStart

MonthsPM$ 1000PeopleDaysDate

MB7.49

72.14661

13.001.46

7/6/97

Life Cycle15.50

115.21105613.006.32

1/1/97

Size39473

ESLOC

MBI 4.2PI 16.0

Risk Analysis - Determinethe Probability of Success

Deadline


When Schedules/Resources are

Fixed - Assess FunctionalityTime Sensitivity to Size

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

Mo

nth

s

30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)

FOC MTTD Sensitivity To Size

8

9

10

11

12

13

Ho

urs

30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)

Uninflated Cost Sensitivity to Size

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

$ (m

illion

s)

30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)

Current Solution Alternative Solutions Acceptable Solution RegionLife Cycle includes R&D, C&T, I&P

TimeEffortUinf CstPk StaffMTTDSize

Life Cycle

31.03259.80

238215.0010.4442000

MonthsPM$ 1000PeopleHoursESLOC

MBI 2.0PI 10.5

TargetSchedule

TargetCost

TargetQuality

Size Range to Test


Staffing Profile

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.50 1 2 3 4 5 6 7 8

Sta

ff

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 *Jan'97

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'98

Feb Mar Apr May Jun Jul Aug

Feas

FD

MB

Maint

0 = FSR

1 = PDR

2 = CDR

3 = FCC

4 = SIT

5 = UOST

6 = IOC

7 = FOC

8 = 99R

RISKTimeEffortUinf CstMin Pk StaffMax Pk StaffFOC MTTD

% 0 10 20 30 40 50 60 70 80 90 100

TimeEffortUinf CstPk StaffMTTDStart

MonthsPM$ 1000PeopleDaysDate

MB8.82

65.33599

10.002.05

8/9/97

Life Cycle18.29

104.34956

10.008.88

1/1/97

Size46473

ESLOC

MBI 3.3PI 16.0

Risk Analysis - Determinethe Probability of Success

Overall Project Risk:Red (High Risk)

Only 45% Probabilityof Meeting Target Schedule

Deadline

Cost and Quality - High Probability

Copyright QSM Associates, Inc.

0

10

20

30

40

50

60

70

80

J an Apr J ul Oct J an Apr J ul Oct

Defects Discovered Each Month High defect rate

Low Mean Time to DefectPoor Quality

Low defect rate High Mean Time to Defect

Good Quality

Reliability ModelingMean Time to Defect


How DevelopmentLifecycles Behave

A Pop Quiz:

“If you tried to shorten the schedule on an application development project by adding staffto say, double (20 people versus 10), how muchwill you able to compress it? Will defects go upor down, and by how much?”


How SoftwareLifecycles Behave

Answer:

Schedules will only compress (nominally) byabout 20 percent.

Defects typically rise by about 6 fold.*

Rule of Thumb: “20/200/6x”

*Source: QSM Industry Database Statistics


Reliability ModelingDefect Severity Categories

Defect Discovery Plan by Category(Expected 50%)

0

20

40

60

80

100

1201 2 3 4 5 6 7 8 9

Defects

1 4 7 10 13 16 19 22 25 28 31 34 *Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul Oct

* Months from beginning of project start month

Critical

Serious

Moderate

Tolerable

Cosmetic

1 = PDR

2 = CDR

3 = FCC

4 = SIT

5 = UOST

6 = IOC

7 = FOC

8 = 99R

9 = 99.9R

5 Severity Categories


Total Defect Rate

0

100

200

300

400

500S 754321

Defects

2 5 8 11 14 17 20 23 26 29 32 35 *Jan'95

Apr Jul Oct Jan'96

Apr Jul Oct Jan'97

Apr Jul Oct Jan'98

Actual

Interpolated

Plan

Green CB

Yellow CB

S = Start

1 = PDR

2 = Bld_1

3 = CDR

4 = Bld_2

5 = TRR

7 = Bld_3

Reliability ModelingReal Data - Actual Vs.

PlannedDefects Starting from

Design


Reliability ModelingReal Data - Actual Vs.

PlannedDefects Starting from Code

Total Defect Rate

0

2

4

6

8

10

12

14

S 75421S 421

Defects

1 5 9 13 17 21 25 29 33 37 *2/4'95

3/4 4/1 4/29 5/27 6/24 7/22 8/19 9/16 10/14

Actual

Interpolated

Plan

Green CB

Yellow CB

S = Start

1 = RB

2 = DD

4 = SIT

5 = UOST

7 = FOC


These reliability drivers are able to be controlled! Reliability and availability is something that we can INFLUENCE!

Reliability ModelingThe Moral of the Story

SizeStaffing

Productivity

Variables Defects


Core Metrics Provide an “Early Warning Indicator”

Gantt Chart

T&E

C&T

R&D

S 1 2 3 4 5 6 7 8S 1 2 3

3 6 9 12 15 18 21 24 27 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Size

0

10

20

30

40

50

60

70S 1 2 3 4 5 6 7 8S 1 2 3

ES

LO

C (th

ou

sa

nd

s)

3 6 9 12 15 18 21 24 27 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Total Cum Cost

0

500

1000

1500

2000

2500

3000

3500S 1 2 3 4 5 6 7 8S 1 2 3

$ (th

ou

sa

nd

s)

3 6 9 12 15 18 21 24 27 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Total Defect Rate

0

20

40

60

80

100S 1 2 3 4 5 6 7 8S 1 2 3

De

fec

ts

3 6 9 12 15 18 21 24 27 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Current Plan Actual Interpolated Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = SIT, 5 = ST, 6 = TRR, 7 = FOC, 8 = 99R

Schedule/MilestonesAppear on Target

Cost is Closeto the Plan

Product is BeingConstructed at a SlowerRate than Planned

Defect Rates areHigher than Planned

Example

Yellow

YellowGreen


Traffic Lights - A VisualTrigger for Course

CorrectionSize

0

10

20

30

40

50

60

70

S 1 2 3 4 5 6 7 8S 1 2 3

ES

LOC

(thousands)

Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul

Actual

Interpolated

Plan

Green CB

Yellow CB

S = Start

1 = HLD

2 = LLD

3 = UIT

4 = SIT

5 = SVT

6 = BT

7 = GA

8 = 99R

Size (ESLOC(K))PI 10.8 9.1 -1.7MBI 1.2 0.6 -0.6

Date 1/2/94 (14.1 mos)

Plan Actual Diff37.12 32.30 -4.82

Data are Consistentlyin the Amber Region

Example


I f W e’re O ff C ourse, W here are W e H eaded?

1 3 5 7 9 11 13 15 17 19 21 23

0

5 0 0 0

1 0 0 0 0

1 5 0 0 0

2 0 0 0 0

2 5 0 0 0

3 0 0 0 0

3 5 0 0 0

4 0 0 0 0

4 5 0 0 0

5 0 0 0 0

E S L O C

M o n th s fro m S ta r t

O rig in a l P la n

7 5 % o f P la n9 0 % o f P la n

1 1 0 % o f P la n1 2 5 % o f P la n

W hich of T hese Ideal C urves has the Best Fit to the A ctual D ata?

E x p e c te d S iz e

4 8 ,0 0 0 E S L O C


Gantt Chart

T&E

C&T

R&D

S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8

3 6 9 12 15 18 21 24 27 30 33 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul

Size

0

10

20

30

40

50

60

70S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8

ES

LO

C (th

ou

sa

nd

s)

3 6 9 12 15 18 21 24 27 30 33 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul

Total Cum Cost

0

500

1000

1500

2000

2500

3000

3500S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8

$ (th

ou

sa

nd

s)

3 6 9 12 15 18 21 24 27 30 33 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul

Total Cum Defects Remaining

0

200

400

600

800

1000

1200S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8

De

fec

ts

3 6 9 12 15 18 21 24 27 30 33 *Oct'92

Jan'93

Apr Jul Oct Jan'94

Apr Jul Oct Jan'95

Apr Jul

Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = SIT, 5 = ST, 6 = TRR, 7 = FOC, 8 = 99R

Adaptive Forecasting: What’s the Remaining “Trajectory”?

Forecasted Schedule:+5.5 Months

Forecasted Cost:+$620K

Forecasted Code Production:Planned PI 10.8, Actual PI 9.1

40 Total Defects WillBe Remaining at theInitial Production Date


Case Study:Case Study:Original Plan vs. Actual DataOriginal Plan vs. Actual Data

Gantt Chart

C&T

S 7654321S 7654321

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Size

0

20

40

60

80

100

S 7654321S 7654321

ES

LOC

(thousands)

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Aggregate Staffing Rate

0

5

10

15

20

25

S 7654321S 7654321

People

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Total Defect Rate

0

10

20

30

40

50

S 7654321S 7654321

Defects

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Total Cum Effort

0

20

40

60

80

100

120

140

S 7654321S 7654321

SM

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Elapsed WeeksSize (ESLOC(K))Agg. StaffTotal Defect RateTotal Cum Effort (SM)PI 19.6 16.5 -3.0MBI 4.8 3.8 -1.0

Date 7/25/2001 (23.53 weeks)

PlanActual/

Forecast Diff23.00 23.00 0.0068.25 58.00 -10.2516.13 15.00 -1.13

6 31 2558.18 59.56 1.37

Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes C&TS = Start, 1 = IDDC, 2 = FDDC, 3 = SIT, 4 = CCUT, 5 = CSIT, 6 = SUOST, 7 = FOC

Actual Data (black squares)

Plan Data Bounds

Forecast Data (white squares)

Example


Gantt Chart

C&T

S 7654321S 7654321

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Size

0

20

40

60

80

100

S 7654321S 7654321

ES

LOC

(thousan

ds)

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Aggregate Staffing Rate

0

5

10

15

20

25

S 7654321S 7654321

People

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Total Defect Rate

0

10

20

30

40

50

S 7654321S 7654321

Defe

cts

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Total Cum Effort

0

25

50

75

100

125

150

S 7654321S 7654321

SM

1 9 17 25 33 41 *2/17'01

4/14 6/9 8/4 9/29 11/24

Elapsed WeeksSize (ESLOC(K))Agg. StaffTotal Defect RateTotal Cum Effort (SM)PI 16.7 16.6 -0.1MBI 3.8 3.7 -0.0

Date 7/29/2001 (24.14 weeks)

PlanActual/

Forecast Diff23.57 23.57 0.0063.12 58.32 -4.8015.00 15.05 0.05

14 29 1561.66 61.66 0.00

Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes C&TS = Start, 1 = IDDC, 2 = FDDC, 3 = SIT, 4 = CCUT, 5 = CSIT, 6 = SUOST, 7 = FOC

Case Study:Case Study:Re-Plan vs. Actual DataRe-Plan vs. Actual Data

Defects currently tracking in high range; probably due to early aggressive schedule

Forecast is on target with plan

Example


Project Office Warboard(Currently in Use by QSM Clients)

Project Sched Cost Size Qlty Ovrll Plan Forecast

InvRptg Jun97 May97

RealFin May97 Apr97

Frcstng Jan97 Aug97

Regtn Jul97 Jun97

PropMgt Aug97 Aug97

*

* underrun ** no data

**

**


In Summary...

It Comes Down to Promises, Commitments, and Expectations!

Understand that SW Dev’t is R&D, and It Behaves That Way! Non-linear interdependencies.

Apply “Negotiation on the Merits” Generate Multiple Options

Test Each Option for Legitimacy, Reasonableness, & Risk

“Know Your Capability” – Estimates Based on History


The Role of Measurement, and Commitments

Estimation& Planning

Control&

Forecasting

Support FutureCommitments

ManageCommitmentCommitment

Analyze Performance on Commitment

HistoryRepository

Assess Viable Strategies

Monitor Status & ReplanPost Project Analysis

Make Commitment


Info Sources on the Web

Software Measurement, Estimation, ControlQSM Associates - www.qsma.com

Information Technology Research PubsCutter Consortium - www.cutter.com/consortium

NegotiationProgram on Negotiation at Harvard Law - www.pon.harvard.edu

Workshops from QSM Associates/Triad Consulting - www.qsma.com/education.html


RecommendedReading - Negotiation

Fisher, Roger and Alan Sharp, “Getting It Done, How to Lead When You’re Not in Charge” HarperCollins 1998.

Fisher, Roger, William Ury and Bruce Patton, “Getting to YES, Negotiating Agreement Without Giving In” Penguin 1981.

Heen, Sheila, Doug Stone and Bruce Patton “DifficultConversations - How to Discuss What Matters Most” Viking/Penguin 1999.


RecommendedReading - Metrics

Carleton, Anita, Park, Robert, and Goethert, Wolfhart , “The SEI Core Measures: Background Information and Recommendations for Use and Implementation” © 1994 The Journal of the Quality Assurance Institute.

Mah, Michael C., “Software Estimation Tricks of the Trade;Secrets They Never Told Me” IT Metrics Strategies© June 2000 Cutter Information Corp.

Putnam, Lawrence H., and Myers, Ware, “Executive Briefing: Controlling Software Development” © 1996 IEEE Computer Society Press.

Tufte, Edward, “Visual Explanations, Images and Quantities,Evidence and Narrative” © 1997 Graphics Press.

Copyright QSM Associates, Inc. 1 Michael C. Mah Managing Partner QSM Associates, Inc. 75 South...

Documents

Transcript of Copyright QSM Associates, Inc. 1 Michael C. Mah Managing Partner QSM Associates, Inc. 75 South...