Www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 8 – Chapters 21-23 1INFO631 Week 8.

www.ischool.drexel.edu

INFO 631 Prof. Glenn Booker

Week 8 – Chapters 21-23

1INFO631 Week 8


Basic Estimation Concepts

Ch. 21

INFO631 Week 8

Slides adapted from Steve Tockey – Return on Software

2


Estimation• Business decisions are always about future

actions– Cash-flow streams for proposal are estimates– Correctness of business decision depends on

accuracy of estimates– Estimation Topics

• Basic Estimation techniques (Ch 21)– What they are and why made

• General Estimation Techniques (Ch 22)– Estimation techniques and why you should use more than one

• Allowing for Inaccuracy in Estimates (Ch 23)– Methods to address inaccuracy

3INFO631 Week 8


Basic Estimation Concepts

Outline – Ch 21

• What is an estimate?

• Estimates and uncertainty

• Cone of uncertainty

4INFO631 Week 8


• An estimate is...

– Basically a quantitative prediction

• Better question might be “why estimate to begin with?

“an assessment of the likely quantitative result.Usually applied to project costs and durationsand should always include some indication of

accuracy (e.g., +/- x percent)”

What is an Estimate?

Reference: [PMI00] 5INFO631 Week 8


Why Use Estimation?

• Business decisions look into future

• Decisions based on factors whose value we might not be know yet– Or can’t be know for certain at the time of the decision– Example

• How much does it cost to develop a product?– Won’t know for sure unless product is developed.

• Goal:– Be close enough to allow you to make the right decision

6INFO631 Week 8


Estimates and Uncertainty

• Estimates imply probability

• Probability function– Set of all outcomes– Not symmetrical– General shape: Nominal Schedule

(50/50 schedule)

7INFO631 Week 8



• Estimate probabilities are not evenly distributed about a mean—there is a shortest possible schedule

• Software projects– Usually have a a lower limit that can not be beat– No definite upper limit.

• Almost always take a little more time and money

• Broad range of possible outcomes– Called Variance (or uncertainty)

• What happens when you arbitrarily adjust an estimate downwards?

8INFO631 Week 8



High Uncertainty

Low Uncertainty

9INFO631 Week 8


Sources of Estimation Uncertainty

• Will the customer want Feature X? • Will the customer want the cheap or expensive version

of Feature X? • If you implement the cheap version of Feature X, will the

customer later want the expensive version after all? • How will Feature X be designed?• How long will it take to debug and correct mistakes made

in implementing Feature X? – Note: DeMarco and Lister coding wars found a 10x variation of

difficulty of implementing designs of the same thing.

10X

10X

10X

10X

10X

10X

10X

10X

10X

10X

10INFO631 Week 8


Exact vs. At-or-less Estimates

• Most decisions don’t need estimates of exact outcomes– Important thing is if actual outcome is at or below estimate– “Did project finish in exactly 10 months?” vs.

“Did project finish in 10 months or less?”• Probability of being at or below any estimate is

probability of hitting that exact outcome plus sum of probabilities of all outcomes less than it– Cumulative probability function—the integral of the probability

function• Note: Impossible zone still applies

Question: Is less always better or worse?

11INFO631 Week 8


Cumulative probability function

• Shape of cumulative probability function is very different than shape of the probability distributions– Since driving 500 miles in exactly 2 hours has a probability of zero then the

probability of getting there in 2 hours or less is also zero.• Probability of getting at or less than the smallest outcome with a non-zero

probability is the same as the probability of hitting exactly that same outcome. • Probability of hitting the second smallest possible outcome or less equals the

probability of hitting exactly the smallest plus the probability of hitting exactly the next-to-smallest.

• The probability of hitting any given outcome or less is the sum of the probabilities of hitting exactly the outcomes from that one all the way back to the beginning (e.g., add up the probabilities of all the arrive-exactly-on times from the given time back to the earliest possible arrival time).

12INFO631 Week 8


Cumulative Probability Functions

P r o b a b i l i t y

Actual arrival time


Actual arrival time


Actual arrival time

13INFO631 Week 8


The 50/50 Estimate


Actual arrival time

50/50 estimate

0.0

0.5

Target during estimation Estimate equally likely to be above or below

50/50 Estimate, Probability = .5 Good target to measure estimation

14INFO631 Week 8


The 50/50 Estimate

From an estimators perspective Target for 50/50 point Estimate equally likely to be over or under

Test of estimators ability See whether the actual outcomes are more than

their estimates as often as they are less. Consistent over/under means there is a

systemic bias in the estimator's estimates

15INFO631 Week 8


Quantifying Estimate Uncertainty

Degree of uncertainty in estimate depends on how much is known Further into a project you have a better

understanding of what is going on Early estimates have more uncertainty that

later of the same thing Cone of Uncertainty - quantifying and showing

how it decreases over time Best to understand by an example

16INFO631 Week 8


Quantifying Estimate Uncertainty

Project: Framus Estimate Act/Est

Scope defined 7 mos 1.43

Requirements complete 11 mos 0.91

Design complete 10 mos 1.00

Code complete 9 mos 1.11

Test complete 9.5 mos 1.05

Project complete 10 mos 1.00

Example – develop a Cone of Uncertainty

17INFO631 Week 8


Relative Error in Estimates

Scopedefined

Requirementscomplete

Designcomplete

Codecomplete

Testcomplete

Projectcomplete

Projectschedule

0.85x

0.9x

1.0x

0.8x

0.6x

1.25x

1.6x

1.15x

1.1x

18INFO631 Week 8


Relative Error in Estimates

Milestone

0

1.0

R e l a t i v e

V a l u e

2.0

Scope defined

Requirements complete

Design complete

Code complete

Testing complete

Actual

.

. . . . ... ..

.

.

.. .. ..

... ..... ....

19INFO631 Week 8


Cone of Uncertainty

Cone of Uncertainty

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

0% 50% 100% 150%

Time

Re

lati

ve

va

lue

Upper limit

Mean

Lower limit

20INFO631 Week 8


So Why Bother with “Cone”

• Knowing “Cone” you can now adjust based on organizational history

21INFO631 Week 8


So Why Bother with “Cone”

Lower Mean Upper

Scope defined 0.40 1.16 1.91

Requirements complete 0.63 1.10 1.56

Design complete 0.73 1.05 1.36

Code complete 0.83 1.00 1.18

Test complete 0.95 1.01 1.07

Project complete 1.0 1.00 1.0

Knowing “Cone” you can now adjust based on organizational history

22INFO631 Week 8


Putting Your Cone of Uncertainty to Work

Milestone

0

20

Weeks

40

Scope defined


Design complete

Code complete

Testing complete

. . . . .

Graph of 50/50 estimates and uncertainty rages

23INFO631 Week 8


“Good” Estimates

Milestone

0

20

Weeks

40

Scope defined


Design complete

Code complete

Testing complete

. . . . .

Actual completion

.

‘Good” Estimates, actual outcome within the uncertainty range

‘Good” = allows decision make to make the right decision

24INFO631 Week 8


Expressing Estimate Uncertainty

• Techniques to express– Plus-or-minus qualifiers– Ranges– Risk quantification– Cases– Coarse dates and time periods– Confidence factors

25INFO631 Week 8


Uncertainty and Ethics

• The IEEE-CS/ACM Software Engineering Code of Ethics and Professional Practices [IEEEACM99] requires software professionals to quote uncertainties:

3.09. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work and provide an uncertainty assessment of these estimates.

26INFO631 Week 8


Cone of Uncertainty in Light of Fixed Schedule

Milestone

F e a t u r e s

Scope defined


Design complete

Code complete

Testing complete

Actual completion

P r i o r i t i z e d

Definitely included

Definitely not included

Uncertain

Over whole project uncertain features will decrease

27INFO631 Week 8


Key Points

• Estimate is “an assessment of the likely quantitative result”

• Estimates are needed to support decisions of factors not known until after the decision is made– Estimates don't have to be perfect

• Estimates are inherently uncertain– That uncertainty can be expressed in a number of different

formats• Your estimates should target the 50/50 point• Cone of Uncertainty shows how uncertainty changes

over time– Measures your ability to commit

28INFO631 Week 8


General Estimation Techniques

Ch. 22

INFO631 Week 8 29

Slides adapted from Steve Tockey – Return on Software


General Estimation Techniques

Outline

• Expert judgment estimation

• Estimation by analogy

• Bottom-up estimation

• Statistical estimation

• Estimation by multiple methods

• Make assumptions explicit

30INFO631 Week 8


Expert Judgment Estimation

• Ask one or more “qualified experts”– Software professional carrying out the decision-making study will

probably be that expert in many cases

• Advantages– Usable even when other techniques aren’t– Useful as a “second opinion”

• Review other estimates– Simplest

• Disadvantages– Most prone to bias and error– Most inaccurate– Depends on having credible “experts”

31INFO631 Week 8


Expert Judgment Estimation

• May want to review expert judgment estimates– Important confidence-building step for all estimation techniques

but particularly valuable here

• If decisions are important enough, converge estimates from several experts– Some organizations choose two-thirds of the way between the

lowest and the highest individual estimates– Nominal group estimate = lowest estimate +

( highest estimate - lowest estimate )*.6667

• Wideband Delphi is another possibility– Choose team, kick-off, individual estimate based on WBS,

review estimates with team, merge together

32INFO631 Week 8


Estimation by Analogy

• Assume thing being estimated is a lot like something we already know about– Base estimate of new thing on experience with

old thing with allowances for differences– Usually more accurate then expert judging

• Of course we’re assuming that there aren’t any other major differences that would influence the cost. – If there were, you’d need to be sure to account

for them too.

33INFO631 Week 8



• Process– Understand the new thing– Get detailed results of one or more analogies– List differences– Estimate impact of each difference– Build up estimate for new thing based on

actual results of analogy project plus the differences

34INFO631 Week 8



• Advantages– Usually more accurate than expert judgment– Relatively quick and easy

• Disadvantages– Requires one or more analogy projects

• Also requires good record keeping

– If not analogous project• Might not be a good choice

35INFO631 Week 8


Bottom-up Estimation

• A popular approach– Aka “module build-up”, “by-engineering procedures”, “top-down”,

“decomposition”, ...

• Developing cash flow streams using a Work Breakdown Structure (WBS) is an application of this estimation technique

• Based on a statistical property called The Law of Large Numbers– Sum of errors in low-level estimates tends to be less than the

error in a single estimate of the whole– In other words, lots of little errors tend to cancel each other

36INFO631 Week 8



• Process– Break whole into smaller parts

• Finer decompositions usually better• Balance with effort to decompose and estimate parts

– Estimate parts• Use appropriate techniques• Some will be overestimated and others underestimated

– Sum estimates of parts• Overestimates tend to cancel underestimates

– If bottom-level estimates don’t address system-wide factors, those will need to be accounted for

37INFO631 Week 8



• Advantage– Law of large numbers tends give more accurate estimates than

single overall estimate

• Disadvantages– Need detailed WBS– May not be practical early in a project– Systemic biases in low level estimates will be compounded– Easy to overlook project- or system-wide factors

• E.g., planning, project management, documentation, testing, rework in a software project

– Effort to create and roll up the detailed estimates may make other methods more attractive

38INFO631 Week 8


Statistical Estimation

• Using an equation relating something countable at decision-time to the factor the decision is based on– Often called “parametric estimation”

• Equation typically derived from historical data, i.e., completed software projects– CoCoMo-II, Price-S, Putnam, SEER-SEM,

SLIM

39INFO631 Week 8


Model User’s Perspective: Cocomo II

KSLOC = Source lines of code (thousands)B = Effort growth factorEM = Effort multiplier

B = Effort growth factorNote: Based on a regression formula

40INFO631 Week 8


Model Developer’s Perspective

• Process:– Identify observable factor(s) that drive the estimate– Gather as many valid data points as possible

• Each point needs quantified factors and actual result• Be certain data is complete and consistent

– Derive functional relationship between factor(s) and observed results

– Optionally, verify that equation is a good predictor• Analyze its ability to match the historical data• This step can be complex and involve heavy-duty statistical

analysis

41INFO631 Week 8


Statistical Estimation• Advantages

– Can be the most accurate– Usually quick and easy to use once the functional relationship has been

derived– Can be as sophisticated as needed– Can be self-tuning

• Disadvantages– Require an adequate base of historical data– Significant effort along with mathematical and statistical knowledge may

be required to derive and verify the functional relationship– Final equation(s) will be sensitive to the quantity, accuracy (consistent

accounting practices across all of the historical data), and relevance of the historical data

– Users may put too much faith in a model that hasn’t been validated

42INFO631 Week 8


Estimation by Multiple Methods

• Sophisticated organizations use several estimation methods throughout the project– “Sanity check” other estimates– Preferred method often changes over the life

of the project• Different methods typically perform better in the

different phases of a project

43INFO631 Week 8


Preferred Estimation Methods by Project Stage

ExpertJudgment

Analogy

Decom-position

Statisticalmethods

Very Early(pre-requirements)

Early(to end of requirements)

Mid(to design complete)

Late(to code complete)

Too early(no design

detail)

Maybe

Reasonable as asecond opinion

Maybe, based on

requirements counts

Yes, feeds into detailed,

low-level estimates

Low level,defect-rework

estimates,reliability models


low-level estimates


low-level estimates


low-level estimates


low-level estimates

44INFO631 Week 8


Make Assumptions Explicit

• Almost impossible to estimate without making assumptions– Those assumptions can have a significant effect on the estimate

• OK to say, for example“I estimate that the Framus project will cost between $0.8 and $1 million and take between 7 and 10 months”

• Much better to say“I estimate that the Framus project will cost between $0.8 and $1 million and take between 7 and 10 months, but this assumes that the control algorithms aren’t more complex than expected and that the project is staffed according to Ron’s projections”

• Making assumptions explicit allows consumers to decide whether or not they agree with the assumptions

45INFO631 Week 8


Make Assumptions Explicit

• Notice that at no point did we try to present an estimate in the form:– “I estimate that the Framus project will cost

$926,364.56 and take 9 months and 8 days.”

• But that’s what people want to hear! – Something very precise, reassuring that we

know exactly what will happen.

INFO631 Week 8 46


Key Points• Expert judgment estimation uses a credible expert• Estimation by analogy creates an estimate from the actual outcome

from one or more known, past situations together with an accounting of the differences

• Bottom-up estimation divides the thing being estimated into smaller parts and builds up the estimate of that whole from the sum of the estimates of the parts– Based on “the law of large numbers”

• Statistical estimation uses equations derived from relevant data• Convergence or divergence among the estimates can indicate how

much trust should be put into those estimates• Preferred estimation technique(s) will probably change over the life

of a software project• Assumptions should be made explicit

47INFO631 Week 8


Allowing for Inaccuracy in Estimates

Ch. 23

INFO631 Week 8 48


Allowing for Inaccuracy

Outline

• Knowledge drives accuracy

• Allowing for inaccuracy

• More effective ways to allow for inaccuracy:– Use ranges of estimates– Sensitivity analysis– Delay final decisions

49INFO631 Week 8


Knowledge Drives Estimation Accuracy

• The more you know, the more accurate your estimates will be

• Hint:– Historical data is easy to collect and incredibly

valuable on later projects…

50INFO631 Week 8


Allowing for Inaccuracy in Estimates

• Allowances for inaccuracy don’t make up for lack of knowledge– Allowances always come at a cost (e.g.,

padding bid reduces chance of selection)

• Ineffective ways to allow for inaccuracy– Increase the MARR for more uncertainty– Shorten the planning horizon

51INFO631 Week 8


Increase the MARR

• Idea: “If the proposal is still profitable even at a higher MARR, then …”

• Counter-example

Year n Project 1 Project 2 0 -$350K -$350K 1 $45K $260K 2 $130K $130K 3 $260K $0 4 $130K $130K

52INFO631 Week 8


Increase the MARR

• “Risk-free MARR”– Doesn’t mean zero risk, just no more than usual risk

• Example – At risk-free MARR = 10%

• PW(i) of Project 1 = $82K• PW(i) of Project 2 = $82K• Outcome – neither project is better than the other

– Assume both projects far more uncertain than usual, Project 2 is more uncertain than Project 1

• All other things equal, Project 1 is safer choice

53INFO631 Week 8


Increase the MARR• Use a risk-adjusted MARR

– Note: Don’t count on risk-adjusted MARRs to address inaccuracy, you could be misled

– 17% for Project 1– 20% for Project 2

• At risk-adjusted MARRs – based on project– PW(17%) of Project 1 = $15K– PW(20%) of Project 2 = $20K

• Contradiction– At risk-free MARR, projects are equivalent and given risk, Project 1

is safer choice– At risk-adjusted MARRs, apparent conclusion is opposite

• Intention of risk-adjusted MARR is to penalize the more uncertain alternative and make it appear less economically attractive

54INFO631 Week 8


Shorten the Planning Horizon

• Idea: “If the proposal is still profitable even with a shorter planning horizon, then …”

• Same counter-example– Shorten planning horizon to 3 years– PW(10%) of Project 1 = -$6K– PW(10%) of Project 2 = -$6K– Neither project is selected

55INFO631 Week 8


More Effective Ways to Allow for Inaccuracy

• Ranges of estimates

• Sensitivity analysis

• Delaying final decisions

56INFO631 Week 8


Case Study - Example• Zymurgenics

– A mythical company that makes process control hardware and software for breweries and wineries

• Potential business opportunity– Certain government regulation is set to expire in exactly 36 months

• Estimates of the main drivers of project cost:– Initial investment = $400,000– Dev environment O&M costs per month = $1000– Development staff costs per month = $35,000– Development project duration = 10 months– Profit per month after development = $40,000

57INFO631 Week 8


Case Study

• Zymurgenics’ MARR is 19.56% annually– Or, 1.5% monthly (according to Tockey’s

rounding!)

PW(1.5%) = -$400K + P/A,1.5,10

( (-$1K + -$35K) * ( 9.2222 ) ) + P/A,1.5,26 P/F,1.5,10

($40K * ( 21.3986 ) * ( 0.8617 ) = $5.6K

58INFO631 Week 8


Ranges of Estimates

• Create 3 separate estimates– Least favorable estimate

• 19 in 20 chance of doing better than (95%)

– Fair estimate• 10 in 20 (even) chance of doing better than

(50/50 break-even)

– Most favorable estimate• 1 in 20 chance of doing better than (5%)

59INFO631 Week 8


Ranges of Estimates

• PW(i) of least favorable estimates = -$830,214• PW(i) of fair estimates = $5,568• PW(i) of most favorable estimates = $652,564

• Chances of all least- or most-favorable estimate happening simultaneously are very remote

Least favorable Fair Most favorable estimate estimate estimateInitial investment $500,000 $400,000 $360,000Operating & maintenance $1500 $1000 $800Development staff cost / month $49,000 $35,000 $24,500Development project duration 15 months 10 months 7 monthsIncome / month $24,000 $40,000 $56,000

60INFO631 Week 8


Sensitivity Analysis

• “Sensitivity” refers to the relationship between the relative change in an estimated factor and the resulting desirability of the alternative– If the estimate for a factor is off by some given percentage, does

the value of the alternative change a little bit or a lot?– Sensitivity analysis shows how much inaccuracies in an

estimate would impact the final decision

• When the various sensitivities are known, the estimator can be careful to get a better estimate for the more sensitive factor(s) and not worry so much about factors with less sensitivity

61INFO631 Week 8



• Basic sensitivity analysis– Repeat for all estimates

• Choose one estimate to vary• Hold all other estimates at fair estimate• Repeat

– Pick a value in the range of the varied estimate– Run the cost function at those values and record the

result

• Until there are no more values to consider

– Until there are no more estimates to consider

62INFO631 Week 8



• Considering change in initial investment

Initial % change in % change investment initial investment PW(i) in PW(i)$360,000 -10% $45,541 722% 380,000 -5 25,541 361 400,000 0 5541 0 420,000 5 -14,459 -361 440,000 10 -34,459 -722 460,000 15 -54,459 -1083 480,000 20 -74,459 -1444 500,000 25 -94,459 -1805

63INFO631 Week 8



• Considering change in operating & maintenance

Operating & % change in % change maintenance O & m PW(i) in PW(i)$800 -20% $7386 33% 900 -10 6464 171000 0 5541 01100 10 4619 -171200 20 3697 -331300 30 2775 -501400 40 1852 -671500 50 930 -83

64INFO631 Week 8



• Considering change in development staff cost

Dev staff % change in % change cost dev staff cost PW(i) in PW(i)$24,500 -30% $102,374 1747% 28,000 -20 70,097 1165 31,500 -10 37,819 582 35,000 0 5541 0 38,500 10 -26,736 -582 42,000 20 -59,014 -1165 45,500 30 -91,291 -1747 49,000 40 -123,569 -2330

65INFO631 Week 8



• Considering change in development duration

Development % change in % change duration dev duration PW(i) in PW(i) 7 mos -30% $204,963 3599% 8 -20 137,497 2381 9 -10 71,028 118210 0 5541 011 10 -58,978 -116412 20 -122,543 -231113 30 -185,169 -344214 40 -246,870 -455515 50 -307,658 -5652

66INFO631 Week 8



• Considering change in income

% change in % change Income income PW(i) in PW(i)$24,000 -40% -$289,475 -5324% 32,000 -20 -141,967 -2662 40,000 0 5541 0 48,000 20 153,049 2662 56,000 40 300,557 5324

67INFO631 Week 8



-50% 50%

-$300K

$300K

Initial investment

Operating & maint

Dev staff cost

Dev durationIncome

Relatively Flat line = insensitive to change, O&M

Steep Slope line = sensitive to change, Income, Dev duration, etc.

68INFO631 Week 8


Delay Final Decisions

• Estimate accuracy is inversely proportional to time between estimate and outcome– Message: cone-of-uncertainty

• Choose to incur somewhat higher costs in the short term to reserve the right to make a major decision when the situation is clearer– Two-Phase Acquisition– Prototyping– Planning checkpoint review [McConnell98]

69INFO631 Week 8


The Cone of Uncertainty

Time

U n c e r t a i n t y

Decisions are usually made when uncertainty is greatest

Delaying the decision significantly reduces

uncertainty

70INFO631 Week 8


Two Phase Acquisition – Application to Software Project

• Phase 1 (10 to 20% of estimated overall project)– Solidify requirements– Identify and address big risks– Plan remainder of project

• Phase 2– If phase 1 passes, continue rest of project

• If a project gets cancelled at the end of Phase 1, is it a failure?

71INFO631 Week 8


Other Practical Approaches

• Planning checkpoint review [McConnell98]– Review project specifics at 20% point

• Business case, Feasibility, etc

– At this point less uncertainty

• Other– Prototyping– Lease versus Buy– Agile development methods

72INFO631 Week 8


Key Points

• Two fairly common approaches are not always effective– Increasing the MARR– Shortening the planning horizon

• Three less common approaches are far more effective– Consider a range of estimates– Sensitivity analysis– Delaying final decisions

73INFO631 Week 8

Www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 8 – Chapters 21-23 1INFO631 Week 8.

Documents

Transcript of Www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 8 – Chapters 21-23 1INFO631 Week 8.