Post on 17-Dec-2014
description
1
Conceptual Design Optimism, Cost and Schedule Growth Effects
Presented at the 2010 NASA Program Management Challenge
9-10 February 2010, Houston, Texas
Claude Freaner, Bob Bitten, Debra Emmons
Used with permission
2
Agenda
• Background/Overview
• Database Description
• Resource Growth– Mass, Power, Cost & Schedule Trends
• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons
• Potential Considerations
3
Background
• Results from a recent study* of 10 NASA missions over the past decade indicated that the average cost and schedule growth of these missions, over and above programmatic reserves, was 76% and 26%, respectively, measured from beginning of Phase B
• One potential causative factor postulated was the inherent optimism in initial concept designs due to competitive pressures
• Inherent optimism can translate to the underestimation of the technical specifications such as mass, power, data rate, and the complexity of a system
• Underestimation of these resources can lead to the underestimation of the final cost of the mission since most cost models use some form of system resources as a predictor of mission cost
• To compound problems, the desire to launch a system as early as possible, in order to obtain science quickly, can lead to a success oriented schedule that may be shorter than historical comparisons would indicate
• This combination of underestimated resources providing an optimistic cost estimate basis combined with a success oriented schedule can contribute to the observed history of cost and schedule growth
* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008
4
Study Approach
• For a set of 20 missions in the study, the mass, power, cost, schedule and other parameters were identified at the beginning of the Preliminary Design phase (NASA Phase B) of a mission*
• These values were then compared to values presented at the Preliminary Design Review (PDR), Critical Design Review (CDR) and at the time of launch to understand the growth over time of each of these resources
• The resource growth is then compared to industry guidelines to understand if these guidelines would have adequately predicted the growth for the mission data set studied
• In addition, resource growth of different mission types and correlation of cost growth in relation to different parameters is also shown
* All parametric and programmatic data were obtained from NASA CADRe data"
5
Agenda
• Background/Overview
• Database Description
• Resource Growth– Mass, Power, Cost & Schedule Trends
• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons
• Potential Considerations
6
Database Description:20 Missions Represent a Wide Range of Recent NASA Missions
• 5 Directed vs. 15 Competed missions
• 7 Planetary missions vs. 13 Earth or near-Earth Orbiters
• 7 Planetary Science vs. 5 Astrophysics vs. 5 Earth Science vs. 3 Heliophysics missions
Planetary? Program Science TypeKey
Center(s)Launch
YearAcquisition
TypeNumber of
Instruments Comments
EO-1 NMP Earth Science GSFC 2000 Competed 5Advanced land imaging technology demonstrator
GENESIS X Discovery Planetary Science JPL 2001 Competed 4Collect samples of solar wind particles at L1 point and return them to Earth
GRACE ESSP Earth Science JPL 2002 Competed 6 Earth Gravity Measurement
SpitzerPhysics of the Cosmos
Astrophysics JPL 2003 Directed 4IR space telescope, the last of the Great Observatories
GALEX Explorers Astrophysics JPL/CalTech 2003 Competed 1 UV space telescope
SWIFT Explorers Astrophysics GSFC 2004 Competed 4 Gamma Ray burst detector
MESSENGER X Discovery Planetary Science APL 2004 Competed 7 Investigate Mercury
MRO X MEP Planetary Science JPL 2005 Directed 7 Investigate history of water on Mars
Deep Impact X Discovery Planetary Science JPL 2005 Competed 3 Comet impactor
Cloudsat ESSP Earth Science JPL 2006 Competed 1 Radar observation of clouds
STEREO STP Heliospheric Science GSFC/APL 2006 Directed 42 spacecraft looking at solar dynamics - Earth leading and trailing orbits
CALIPSO ESSP Earth Science LARC 2006 Competed 3 Aerosols
New Horizons
XNew Frontiers
Planetary Science APL 2006 Competed 7 Investigate Pluto
DAWN X Discovery Planetary Science JPL 2007 Competed 2 Investigate Ceres and Vesta protoplanets
AIM Explorers Heliospheric Science LASP 2007 Competed 3 Aeronomy of Ice in Mesosphere
Fermi (GLAST)
Physics of the Cosmos
Astrophysics GSFC 2008 Directed 2 Gamma Ray Telescope
IBEX Explorers Heliospheric Science GSFC 2008 Competed 2Interaction between solar wind and interstellar medium
Kepler Discovery Astrophysics JPL 2009 Competed 1 Search for Earth-sized exoplanets
LRO XRobotic Lunar
ESMD/Planetary Science
GSFC 2009 Directed 7 Origin of the Moon
OCO ESSP Earth Science JPL 2009 Competed 1Carbon Dioxide Investigation. Mission failed due to launch vehicle failure
7
Updated 20 Mission Results Are Similar to Initial 10 Mission Paper* Results
43% 42%
76%
36%37%41%
56%
38%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Mass Power Cost Schedule
Aver
age P
erce
nt G
rowt
h
10 Mission Study20 Mission Study
1 1 2 2
Note: 1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves
* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008
8
Agenda
• Background/Overview
• Database Description
• Resource Growth– Mass, Power, Cost & Schedule Trends
• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons
• Potential Considerations
9
37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Mission #1
Mission #2
Mission #3
Mission #4
Mission #5
Mission #6
Mission #7
Mission #8
Mission #9
Mission #1
0
Mission #1
1
Mission #1
2
Mission #1
3
Mission #1
4
Mission #1
5
Mission #1
6
Mission #1
7
Mission #1
8
Mission #1
9
Mission #2
0
Averag
e
Sate
llite
Mas
s G
row
th R
elat
ive
to C
BE
(%)
Summary of Mass Growth from Start of Phase B
Mass Growth Exceeds Typical Guidance:13 out of 20 Missions Exceed 30% Growth from Current Best Estimate
Planetary MissionsAverage
Earth Orbiting Missions
10
Summary of Satellite Mass Growth Over Time
Majority of Mass Growth Occurs Primarily Prior to PDR
37%
0%
28%
19%
-10%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
ATP PDR CDR Launch
Sate
llite
Mas
s G
row
th o
ver C
BE
(%)
Mission #1Mission #2Mission #3Mission #4Mission #5Mission #6Mission #7Mission #8Mission #9Mission #10Mission #11Mission #12Mission #13Mission #14Mission #15Mission #16Mission #17Mission #18Mission #19Mission #20Average
11
41%
-20%
0%
20%
40%
60%
80%
100%
120%
140%
Mission #1
Mission #2
Mission #3
Mission #4
Mission #5
Mission #6
Mission #7
Mission #8
Mission #9
Mission #1
0
Mission #1
1
Mission #1
2
Mission #1
3
Mission #1
4
Mission #1
5
Mission #1
6
Mission #1
7
Mission #1
8
Mission #1
9
Mission #2
0
Averag
e
Sate
llite
Pow
er G
row
th R
elat
ive
to C
BE
(%)
Summary of Power Growth from Start of Phase B
Power Growth Exceeds Typical Guidance:8 out of 19 Missions Exceed 30% Growth from Current Best Estimate
Planetary MissionsAverage
Earth Orbiting Missions
12
Summary of Power Growth Over Time
Majority of Power Growth Occurs Primarily Prior to CDR
19%
36%41%
0%
-20%
0%
20%
40%
60%
80%
100%
120%
140%
ATP PDR CDR Launch
Sate
llite
Pow
er G
row
th o
ver C
BE
(%)
Mission #1Mission #2Mission #3Mission #4Mission #5Mission #6Mission #7Mission #8Mission #9Mission #10Mission #11Mission #12Mission #13Mission #14Mission #15Mission #16Mission #17Mission #18Mission #19Mission #20Average
13
Summary of Cost Growth from Start of Phase B
Development Cost Growth is Significant:Average Cost Growth, Over & Above Reserves, is 56%
56%
0%
20%
40%
60%
80%
100%
120%
140%
160%
Mission #1
Mission #2
Mission #3
Mission #4
Mission #5
Mission #6
Mission #7
Mission #8
Mission #9
Mission #1
0
Mission #1
1
Mission #1
2
Mission #1
3
Mission #1
4
Mission #1
5
Mission #1
6
Mission #1
7
Mission #1
8
Mission #1
9
Mission #2
0
Averag
e
Dev
elop
men
t Cos
t Gro
wth
ove
r Bas
elin
e (%
)
$-
$40
$80
$120
$160
$200
$240
$280
$320
$360
$400
Dev
elop
men
t Cos
t Gro
wth
ove
r Bas
elin
e ($
M)
Planetary MissionsAverage
Earth Orbiting Missions
Absolute Growth in $M
14
Summary of Cost Growth Over Time
Cost Growth Occurs Primarily After PDR:More than Half of Growth (14% to 56%) is Realized After PDR
21%
56%
14%
0%
-25%
0%
25%
50%
75%
100%
125%
150%
ATP PDR CDR Launch
Mis
sion
Dev
elop
men
t Cos
t Gro
wth
ove
r Bas
elin
e w
ith R
eser
ves
(%)
Mission #1Mission #2Mission #3Mission #4Mission #5Mission #6Mission #7Mission #8Mission #9Mission #10Mission #11Mission #12Mission #13Mission #14Mission #15Mission #16Mission #17Mission #18Mission #19Mission #20Average
15
Summary of Schedule Growth from Start of Phase B
Schedule Growth is Significant:Average Schedule Growth is on the order of 18 Months
38%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Mission #1
Mission #2
Mission #3
Mission #4
Mission #5
Mission #6
Mission #7
Mission #8
Mission #9
Mission #1
0
Mission #1
1
Mission #1
2
Mission #1
3
Mission #1
4
Mission #1
5
Mission #1
6
Mission #1
7
Mission #1
8
Mission #1
9
Mission #2
0
Averag
e
Dev
elop
men
t Sch
edul
e G
row
th o
ver B
asel
ine
(%)
0
5
10
15
20
25
30
35
40
45
50
Dev
elop
men
t Sch
edul
e G
row
th o
ver B
asel
ine
(Mon
ths)
Planetary MissionsAverage
Earth Orbiting Missions
Absolute Growth in Months
16
Summary of Schedule Growth Over Time
Schedule Growth Occurs Primarily After PDR:More than Half of Growth (13 of 18 Months) is Realized After PDR
10%17%
38%
-20%
0%
20%
40%
60%
80%
100%
ATP PDR CDR LaunchMis
sion
Dev
elop
men
t Sch
edul
e G
row
th o
ver P
lan
(%)
Mission #1Mission #2Mission #3Mission #4Mission #5Mission #6Mission #7Mission #8Mission #9Mission #10Mission #11Mission #12Mission #13Mission #14Mission #15Mission #16Mission #17Mission #18Mission #19Mission #20Average
5 months 8 months18 months
17
Agenda
• Background/Overview
• Database Description
• Resource Growth– Mass, Power, Cost & Schedule Trends
• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons
• Potential Considerations
18
Cost & Schedule Still Show Significant Growth from Baseline Established at PDR
37%41%
38%
15%19%
37%
27%
56%
0%
10%
20%
30%
40%
50%
60%
Mass Power Cost Schedule
Aver
age P
erce
nt G
rowt
h
From Phase B StartFrom PDR
1 1 2 2
Note:1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves
Over Half of Uncertainty for Mass & Power is Retired by PDR while 2/3 of Cost & Schedule Uncertainty Remain
19
Data Shows that Programmatic Baseline Maturity Lags Technical Design Maturity
0%
10%
20%
30%
40%
50%
60%
Phase B Start PDR CDR Launch
Aver
age P
erce
nt G
rowt
h
MassPowerCostSchedule
Technical Design Mature by CDR
Programmatic RealizationNot Until Launch
20
Comparison of Competed vs. Directed Missions: No Significant Difference in Results
40%
29%
51%
31%36%
42%
58%
40%
0%
10%
20%
30%
40%
50%
60%
70%
Mass Power Cost Schedule
Aver
age
Perc
ent G
rowt
h fro
m P
hase
B S
tart Directed (5)
Competed (15)
1 1 2 2
Note:1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves
Growth in All Resources are Similar Between Competed & Directed Missions
21
Planetary vs. Earth Orbiting Missions: Resource Growth is Similar for Mass & Power
40% 37%
25%22%
35%40%
73%
47%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Mass Power Cost Schedule
Aver
age
Perc
ent G
rowt
h fro
m P
hase
B S
tart Planetary (7)
Earth Orbiting (13)
1 1 2 2
Note:1) As measured from Current Best Estimate, not including reserves2) As measured from baseline estimate, including reserves
Cost & Schedule Growth for Planetary Missions is Significantly Less
22
Correlation Between Cost and Schedule Growth Indicates that Schedule Growth Has Strong Influence on Cost Growth
% Cost Growth = 1.4775 * % Schedule GrowthR2 = 0.6166
0%
20%
40%
60%
80%
100%
120%
140%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schedule Growth from Phase B Start
Dev
elop
men
t Cos
t Gro
wth
from
Pha
se B
Sta
rt
23
Payload Mass and Cost Growth Significantly Greater than Spacecraft Mass & Cost Growth
60%
101%
33%44%
0%
20%
40%
60%
80%
100%
120%
Mass Cost
Aver
age
Perc
ent G
rowt
h fro
m P
hase
B S
tart Payload
Spacecraft
1 1
Note:1) As measured from Current Best Estimate, not including reserves
Data Indicated Payload Resource has Greater Uncertainty than Spacecraft
24
Cumulative Cost Distribution Shows Wide Distribution of Payload Cost Growth vs. Spacecraft Cost Growth
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
-25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250%
Cost Growth from Phase B Start over Initial Estimate without Cost Reserve
Cum
ulat
ive
Dis
trib
utio
n fo
r All
Dat
a Po
ints
Spacecraft Cost GrowthPayload Cost Growth
Payload Cost Growth hasMuch Greater Variancethan Spacecraft Growth
25
Instrument Payload Mass/Cost Growth Data Over Time Also Indicates Payload Cost Lags Payload Maturity
0%
31%43%
60%
0%
18%
37%
101%
0%
20%
40%
60%
80%
100%
120%
Phase B Start PDR CDR Launch
Perc
ent G
rowt
h ov
er C
BE w
/o re
serv
es
Payload MassPayload Cost
Payload Cost Significantly Lags Payload Design Maturity
26
Range of Mass Growth at Instrument & Subsystem LevelCan Provide Guidance for Initial KDP-B Cost Estimates
Range of Mass Growth is Large and Exceeds Typical Industry Guidance
-75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% 275%
Struct & Mech
Power
C&DH
Comm
Thermal
ADCS
Propulsion
Instruments
Range for Mass Distributions
Below Mean
Above Mean
Typical Mass ReserveDesign Guidance 30%
Subsystem Min Mean Max MedianInstrument 11% 60% 202% 51%Propulsion -73% 10% 69% 18%ADCS -21% 29% 108% 24%Thermal -36% 77% 269% 58%Comm -55% 5% 76% 7%C&DH -37% 16% 104% 6%Power -37% 45% 103% 40%Structure & Mech -19% 57% 142% 60%
27
Agenda
• Background/Overview
• Database Description
• Resource Growth– Mass, Power, Cost & Schedule Trends
• Findings– Progression from milestones– Comparison of mission types– Payload vs. Spacecraft comparisons
• Potential Considerations
28
Potential Considerations• For Project Managers
– Mass and power reserve guidelines for spacecraft and payload could be increased to be more consistent with historical averages
– Schedule growth seems to be a key factor in controlling cost growth
– More emphasis could be placed on early payload designs as there is much greater uncertainty in the payload development than the spacecraft
• For Cost Analysts– Wider ranges of input parameters could be used to provide more
robust initial cost estimates which can address uncertainty in early design
29
References
1) Freaner C., Bitten R., Bearden D., and Emmons D., “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, 2008 SSCAG/SCAF/EACE Joint International Conference, Noordwijk, The Netherlands, 15-16 May 2008.
2) Bitten R., Emmons D., Freaner C., “Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines”, IEEE Aerospace Conference, Big Sky, Montana, March 3-10, 2007.
3) Bitten R.E., “Determining When A Mission Is "Outside The Box": Guidelines For A Cost-Constrained Environment”, 6th IAA International Low Cost Planetary Conference, October 11-13, 2005.
4) Bitten R.E., Bearden D.A., Lao N.Y. and Park, T.H., “The Effect of Schedule Constraints on the Success of Planetary Missions”, 5th IAA International Conference on Low-Cost Planetary Missions, 24 September 2003
5) Emmons D., “A Quantitative Approach to Independent Schedule Estimates of Planetary & Earth-orbiting Missions”, 2008 ISPA-SCEA Joint International Conference, Netherlands, 12-14 May 2008