Cost Estimation of Human Systems Integrationseari.mit.edu/documents/presentations/Liu_SM_PPT.pdf ·...
Transcript of Cost Estimation of Human Systems Integrationseari.mit.edu/documents/presentations/Liu_SM_PPT.pdf ·...
Cost Estimation of Human Systems
Integration
1stLt. Kevin Liu, USMCSM in Technology & Policy, June 2010
Research Advisors: R. Valerdi and D.H. Rhodes
The views expressed in this presentation are those of the author and do not reflect the official policy or position of the United States Air Force, Marine Corps, Department of Defense, or the U.S. Government.
This research was generously supported by the Air Force Human Systems Integration Office.
Motivation
seari.mit.edu © 2010 Massachusetts Institute of Technology 2
Research Questions/Hypothesis
seari.mit.edu © 2010 Massachusetts Institute of Technology 3
R1: How can the “right” amount of effort to invest in HSI be determined?
R2: How much does HSI effort cost?
R3: What is the relationship between HSI and systems engineering?
Hypothesis: HSI effort can be estimated as a function of Systems Engineering Effort.
Research Outline
seari.mit.edu © 2010 Massachusetts Institute of Technology 4
1. Review of Relevant Topics- Defense Acquisition- Systems Engineering
Hypothesis 1: HSI effort can be estimated as a function of SE.
2. Problem Formulation- HSI and Cost Estimation- Case Study: F119 Engine- HSI and COSYSMO
Hypothesis 2: HSI effort can be estimated by counting HSI-related requirements
3. Implementation and ValidationWorkshop 1: How can COSYSMO
requirements counting rules better account for HSI?
Workshop 2: How can an IPT reach a consensus about effort of HSI-related requirements?
4. Conclusions, Contributions, Recommendations
Review of Relevant Topics
seari.mit.edu © 2010 Massachusetts Institute of Technology 5
Image adapted from DoD 5000.02: Operation of the Defense Acquisition System
Stakeholder Requirements Definition
Requirements Analysis, Architecture Design
Implementation, Integration, Verification
Transition, Validation
Systems Engineering Technical Processes
Is a new program necessary?
Defense Acquisition Management
Problem Formulation-HSI
seari.mit.edu © 2010 Massachusetts Institute of Technology 6
HSI is tightly coupled to Systems Engineering
HSI came from Human Factors
““a comprehensive management and technical approach for addressing the human element in weapon system development and
acquisition,” (Air Force Instruction 10-601)
“HSI: interdisciplinary technical and management processes for
integrating human considerations within and across all system
elements; an essential enabler to systems engineering practice.”
INCOSE
HSI cost studies look at specifics, not prediction.
Bias & Mayhew - UsabilityDray & Karat - User Interface
Hypothesis: HSI effort can be estimated as a function of SE effort and used for predictive
purposes.
Bosert - HFEBooher (1997) –
Several Army Systems
Problem Formulation-Case Study: F119 Engine
seari.mit.edu © 2010 Massachusetts Institute of Technology 7
Research Questions1.How did Pratt & Whitney determine how much HSI effort would be needed?2.How much did HSI effort eventually cost?3.How did HSI fit into the larger SE picture?
Proposition: HSI effort can be isolated from total SE effort spent.
Takeaways1.HSI cost cannot be isolated from SE cost. 2.Requirements drive effort/focus spent on HSI3.Teams are important to SE/HSI process
Methodology-Case Study Research (Yin 2009)-Unit of Analysis: Early development at P&W- Method: Interviews and Literature- Interpretation of results: proposition vs. rival proposition Conclusion: Rival proposition supported.
Image: Pratt & Whitney
Problem Formulation-HSI and COSYSMO
seari.mit.edu © 2010 Massachusetts Institute of Technology 8
COSYSMO
SizeDrivers
EffortMultipliers
Effort
Calibration
rementscesriosthms
- Application factors-8 factors
- Team factors-6 factors
Hypothesis 2: HSI effort can be estimated by counting requirements related to HSI.
RequirementsInterfacesAlgorithmsOperational scenarios
Determine System of Interest
Tested?Verified?Designed?
Yes
No
1
2
Sketch System of Interest’s
Relationship to Rest of System
3
4Assess Complexity
Shall’sWill’sMust’s
5
Workshop 1: Application of Requirements Counting Rules
seari.mit.edu © 2010 Massachusetts Institute of Technology 9
Research Question: How can COSYSMO counting rules be modified to improve counting of HSI-related requirements?
Workshop Hypothesis #1:Using the cost estimation counting rules will produce requirements counts with high reliability across respondents.
Workshop 1: Application of Requirements Counting Rules
seari.mit.edu © 2010 Massachusetts Institute of Technology 10
Research Question: How can COSYSMO counting rules be modified to improve counting of HSI-related requirements?
Workshop Hypothesis #2: The cost estimation counting rules will help users quantify the number of HSI requirements to be input into COSYSMO.
Needs Identified
1.Better tools for consensus-building.
2.Further exploration of the impact of HSI on effort.
Workshop 2: Estimation of Relative HSI Effort
seari.mit.edu © 2010 Massachusetts Institute of Technology 11
Research Question: How can an IPT reach a consensus about the effort associated with an HSI-related requirement during early cost estimation?
1x 2x 4x 8x
“What is the equivalent effort of the HSI-related requirement in question, as compared
to a nominal requirement?”
1. Complete Round 1 individuallya) Review and discuss
2. Round 2 of estimationa) Review and discuss
3. Round 3 of estimation
Building Consensus
“usually I think of training as something that you worry about after the system’s already
developed or kind of set, so you’re really not adding effort”
“[training] touches so many other folks and it has such a large human element in it, you really got to give yourself a lot longer schedule, and not unusually a lot more money”
Workshop 2: Estimation of Relative HSI Effort
seari.mit.edu © 2010 Massachusetts Institute of Technology 12
Consensus improved in 9 out of 10 cases
Conclusions and Recommendations
seari.mit.edu © 2010 Massachusetts Institute of Technology 13
• Modifications recommended for COSYSMO counting rules
• Disconnect between industry perception of HSI and HSI in DoD requirements
• Nonfunctional requirements must be considered early to avoid requirements creep
• Methodological contribution: quantification of Wideband Delphi results.
• Maturing understanding of how HSI effort is planned, tracked, and executed as part of SE in Air Force and industry