RESEARCH PROFILE Systems Engineering Economics
Transcript of RESEARCH PROFILE Systems Engineering Economics
RESEARCH PROFILE
Systems Engineering Economics
October 16, 2007
Dr. Ricardo Valerdi
Massachusetts Institute of Technology
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 2
Research Cluster-Portfolio
Mapping
XSE Strategic
Guidance
XSE in the
Enterprise
XSE Economics
XDesigning for
Value
Robustness
XXXXSocio-Tech
Decision
Making
SE-SynthesisSE-FieldR-STARSV-STARS
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 3
Research Portfolio (3)
SYSTEMS ENGINEERING ECONOMICS
This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering
value to stakeholders. Examples include:
• Measurement of productivity and quantifying the ROI of systems engineering
• Advanced methods for reuse, cost modeling, and risk modeling
• Application of real options in systems and enterprises
• Leading indicators for systems engineering effectiveness
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 4
Outline
• Motivation for economics in systems engineering
• Economic Principles – Cost
– Diseconomies of scale
– Productivity
– Consumer behavior
– Judgment and decision making
– Risk
– Reuse
– Value and Return on Investment
– Leading Indicators
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 5
Motivation
Daniel Kahneman
• Psychologist (Princeton University)
• Nobel laureate in Economics (2002)
Barry Boehm
• Computer Scientist (USC)
• Father of Software Engineering Economics
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 6
Definitions
Systems Engineering• An interdisciplinary approach and means to enable the realization of successful systems. It focuses on defining customer needs and required functionality early in the development cycle, documenting requirements, then proceeding with design synthesis and system validation while considering the complete problem.
Economics• The social science that deals with the production, distribution, and consumption of goods and services and with the theory and management of economies or economic systems.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 7
All models are wrong…
…but some of them are useful.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 8
Systems Engineering Knowledge
Hierarchy
�Prediction
�Symbolic
Manipulation
�Symbolic
Representatio
n
��Quantification
&
Measurement
����Abstraction
�����Classification
����Observation
VBSSE[2
] (Jain &
Boehm
2006)
Ontologies
(Honour &
Valerdi
2006)
GUTSE[1]
(Friedman
2004)
DoD
Architecture
Framework
(DODAF
2004)
COSYSMO
(Valerdi et
al 2003)
Maturity
Models
(CMMI
2002)
SE
Standards
(ANSI/EIA
1999,
ISO/IEC
2002)
Vee Model
(Forsberg &
Mooz 1995)
Systems
Architecting
heuristics
(Rechtin
1991)
[1] Grand Unified Theory of Systems Engineering[2] Value-Based Systems & Software Engineering
Dixit, I., Valerdi, R., “Challenges in the Development of Systems Engineering as a Profession,” INCOSE Symposium, San Diego, CA, June 2007.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 9
Outline
• Motivation for economics in systems engineering
• Economic Principles – Cost
– Diseconomies of scale
– Productivity
– Consumer behavior
– Judgment and decision making
– Risk
– Reuse
– Value and Return on Investment
– Leading Indicators
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 10
COSYSMO
SizeDrivers
EffortMultipliers
Effort, $
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
+3 Volatility Factors
- Application factors-8 factors
- Team factors-6 factors
Valerdi, R., Systems Engineering Cost Estimation with COSYSMO, Wiley, 2008.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 11
Where:
PMNS = effort in Person Months (Nominal Schedule)
A = calibration constant derived from historical project data
k = {REQ, IF, ALG, SCN}
wx = weight for “easy”, “nominal”, or “difficult” size driver
= quantity of “k” size driver
E = represents diseconomy of scale
EM = effort multiplier for the jth cost driver. The geometric product results in an
overall effort adjustment factor to the nominal effort.
xΦ
COSYSMO Algorithm
∏∑=
⋅
Φ+Φ+Φ⋅=
14
1
,,,,,, )(j
j
E
k
kdkdknknkekeNS EMwwwAPM
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 12
Diseconomies of scale
(i.e., Non-linear effects of size)
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 13
Productivity
Y-intercept = 300 PM (25 people for 1 year)
Productivity = [0.004 min, 0.22 max] (Size/SE_Hr)
Range = Size[374 min, 8743 max]; SE_Hours[10,992 min, 1,749,372 max]
“x” = SW
“+” = HW
0
0.05
0.1
0.15
0.2
0.25
1 2 3 4 5 6 7 8
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 14
Does size matter?
ADJHRSSE
SizeP
__=
• Evaluated by productivity, P, as a function of Size & Effort
• Projects with extremely low and extremely high productivity introduced new questions
∑ ⋅+⋅+⋅+⋅= OPSCALGINTFREQSize ωωωω
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 15
Feasibility Plans/Rqts. Design Develop
and TestPhases and Milestones
Relative
Size
Range
Operational
Concept
Life Cycle
Objectives
Life Cycle
Architecture
Initial
Operating
Capability
x
0.5x
0.25x
4x
2x
Estimation Accuracy
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 16
Bayesian Approximation
1.06
Literature,
behavioral analysis
A-priori
Experts’ Delphi
Noisy data analysis
A-posteriori Bayesian update
Productivity Range =Highest Rating /Lowest Rating
1.451.51
1.41
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 17
Outline
• Motivation for economics in systems engineering
• Economic Principles – Cost
– Diseconomies of scale
– Productivity
– Consumer behavior
– Judgment and decision making
– Risk
– Reuse
– Value and Return on Investment
– Leading Indicators
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 18
0
2
4
6
8
10
12
14
16
18
20
300 700 1100 1500 1900 2300 2700 More
Person Months
Frequency
0
5
10
15
20
25
200
385.71
4285
757
1.42
8571
475
7.14
2857
194
2.85
7142
911
28.571
429
1314
.285
714
More
Person Months
Frequency
0
5
10
15
20
25
30
35
40
150 300 450 600 750 900 1050 More
Person Months
Frequency
Intuitive Judgments
SE Effort given
Conceptualize
took 300 PM
SE Effort given
Conceptualize &
Develop
took 300 PM
SE Effort given
Conceptualize,
Develop, OT&E
took 300 PM
1386=µ 594=µ 390=µ
Valerdi, R., “Cognitive Limits of Software Cost Estimation,” 1st Conference on Empirical Software Engineering & Measurement, September 2007, Madrid, Spain.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 19
594=µ 390=µ
Cone of Uncertainty
SE Effort given
Conceptualize
took 300 PM
SE Effort given
Conceptualize &
Develop
took 300 PM
SE Effort given
Conceptualize,
Develop, OT&E
took 300 PM
1386=µ
758=σ 241=σ 145=σ
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 20
What do decision makers trust?
0
5
10
15
20
25
90 94 98 102
106
110
114
118
122
126
130
Person Months
Frequency
0
5
10
15
20
25
30
900
940
98010
2010
6011
0011
4011
8012
2012
6013
00
Person Months
Frequency
110=µ
COSYSMO = 1,000 PM
Historical data = 1,100 PM
1122=µ
COSYSMO = 100 PM
Historical data = 110 PM
9=σ 111=σ
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 21
Person Months Risk
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
0 25000 50000 75000 100000
Person Months
Risk (= Prob. That Actual Person Months
Will Exceed Indicated, X-Axis, Figure)
Person Months Confidence (Cumulative Probability)
0%5%10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%85%90%95%100%
0 25000 50000 75000 100000
Person Months
Cumulative Probability of Person
Months
Risk/Confidence Plots
Valerdi, R., Gaffney, J., “Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers: Some Techniques for Enhancing Accuracy,”
5th Conference on Systems Engineering Research, March 2007, Hoboken, NJ.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 22
Systems Engineering Risk
Assessment
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 23
Reuse in Systems Engineering
• Let:
– NN=count of new requirements;
– NM=count of modified requirements;
– NR=count of reused requirements;
– ND=count of deleted requirements
• Then, NT=total count of
requirements=NN+NM+NR+ND
• Define: ET=NN+(cM*NM)+(cR*NR)+(cD*ND);
– Where: cM, cR, cD, are the unit effort values for a
modified, a reused, or a deleted requirement
respectively, relative to that for a new requirement.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 24
Incremental Software Systems
Engineering ROI
0
2
4
6
8
10
Incremental RESL Improvement Investment
10,000KSLOC
RO
I
L N N H H VH VH EHVL L-2
1,000KSLOC
100KSLOC
10KSLOC
Boehm, B., Valerdi, R., Honour, E., “The ROI of Systems Engineering,” Systems Engineering, 2008. (in press)
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 25
How Much Architecting is
Enough?
0
20
40
60
80
100
0 10 20 30 40 50
% time added for architecture and risk resolution
% t
ime
ad
ded
to
overa
ll s
ch
ed
ule
10,000KSLOC
X
X
X
X
X
X
X
X
X
X
X
XX
XX
X
% of project schedule devoted to initial architecture and risk resolution
Added schedule devoted to rework (COCOMO II RESL factor)
Total % added schedule
+
+
+
+
++
Sweet Spot
++
+ ++
++
++
+
100KSLOC
10KSLOC
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 26
Systems Engineering Leading
Indicators
• Requirements Trends
• System Definition Change Backlog Trend
• Interface Trends
• Requirements Validation Trends
• Requirements Verification Trends
• Work Product Approval Trends
• Review Action Closure Trends
• Risk Exposure Trends
• Risk Handling Trends
• Technology Maturity Trends
• Technical Measurement Trends
• Systems Engineering Staffing & Skills Trends
• Process Compliance Trends
Requirements Trends
TIME
Requirements Growth Trends
TIME
NUMBER OF REQUIREMENTS
JulyMar Apr May JuneFebJan
LEGEND
Planned Number
Requirements
Actual Number
Requirements
Aug Sep Oct Nov Dec
Projected Number
Requirements
SRR PDR CDR ….
Corrective
Action Taken
Rhodes, D., Roedler, G., Systems Engineering Leading Indicators Guide, V. 1.0, June 15, 2007
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 27
Gelperin, D., New Approaches to Specifying Clear Testable System Requirements, Requirements Quarterly, RQ36, pp. 17, June 2005.
© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 28
Summary
• Systems Engineering Economics– developing a new paradigm that encompasses an
economics view of systems engineering to achieve
measurable and predictable outcomes while
delivering value to stakeholders.
• Next segment– Research Report: Tsoline Mikaelian, Managing
Uncertainty in Complex Systems and Enterprises
using Real Options
– Brief Overview: Chris Roberts, Harmonizing Systems
Engineering of Enterprises and their Context