Post on 13-Aug-2020
Why Johnny Can’t Make Good Decisions and What We Can
Do About It C. Robert Kenley
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012
Introduction • The purpose of this panel is to illuminate
the importance and impact of early program decisions
• My purpose is to illuminate the – Loss of focus in our community on making
good decisions early in a program – Evidence that we are poised to re-focus our
community on this important aspect of systems engineering
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 2
A Digression: The Original Johnny and His Problem
1955: A problem is identified and a solution is proposed 1981: The problem persists
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 3
WHAT HAPPENED WHEN JOHNNY GREW UP AND BECAME A SYSTEMS ENGINEER?
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 4
1990: US DoD SE Process Includes Decision Making as a Key Step
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 5
DSMC, Systems Engineering Management Guide, January 1990
1991: Forsberg and Mooz Innocently Introduce the Vee-Model
The Original Vee-Model Accompanying SE Process Included Decision Making
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 6
Forsberg and Mooz, Proceedings of the First Annual Symposium of NCOSE, Chattanooga, TN, October 1991
Decision-Making Step
1991: Forsberg and Mooz Describe How the Vee-Model and the SE Process (with Decision Making) Fit Together
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 7
Forsberg and Mooz, Proceedings of the First Annual Symposium of NCOSE, Chattanooga, TN, October 1991
2000: Sheard Identifies Different Types Of Systems Engineering
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 8
Sheard, S. A., Proceedings of the Tenth International Symposium of INCOSE , Minneapolis, MN, July 2000.
Com
plex
ity
Life Cycle
Type 3 Approach Any kind of engineering
Type 2 Program SE Unprecedented Solutions
Type 1 Discovery Unprece- dented Problems
My Hypothesis • The Program SE cadre implicitly adopted the Vee-Model as their
systems engineering process – Sophisticated decision-making methods were not always needed and
were not highly valued – Program SE developed and integrated components into unprecedented
solutions after the system requirements and solutions were defined – Johnny had lots of fun doing other technical work that kept him busy
• The Discovery cadre found employment elsewhere – Sophisticated decision making for unprecedented problems was the
main thing – Discovery SE produced the tradeoffs between problem solutions,
performance, costs, and schedule, and justified the selection of an solution to be developed by Program SEs
– Jane (Johnny’s cousin) worked as a decision analyst for the Discovery SE industry and had lots of fun doing applied mathematics that kept her busy
9
Johnny’s Framework in One Chart
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 10
Occasional Decision Making When We Must
From Haskins, C., ed.. 2011. Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities. Version 3.2.1 Revised by M. Krueger, D. Walden, and R. D. Hamelin. San Diego: INCOSE
Jane’s Framework in One Chart (All Decision-Making, All the Time)
Known alternatives
Alternative Design
Multiple known alternatives
Multiple alternative designs
Strategy
National Policy
International Policy
None
Decision
Decision Maker & Stakeholder Interaction
Representatives
Single Decision Maker
External Decision Reviews
Decision Conference
Dialog Decision Process
Single objective
Multiple objective (Direct Assessment)
Multiple Objective Value Functions
Value-Focused Thinking
Value Preference
Importance Weights
Swing Weights
Swing Weight Matrix
Uncertainty
New Scenarios
Existing Scenarios
None Probability (Multiple Experts)
Probability (Single Expert)
None
No Weights
Risk Preference
Time Preference
None
Discounting
Single objective Utility
Expected Value
Multiple Objective Utility
Monte Carlo Simulation Decision Trees Influence Diagrams Bayesian Nets
Not recommended.
Strategy Generation Table
Decision Opportunity
Parnell, G. S., Decision Analysis in One Chart, Decision Line, Newsletter of the Decision Sciences Institute, May 2009 © Copyright 2011 Gregory S. Parnell. All rights reserved.
WHAT HAVE WE BEEN DOING ABOUT JOHNNY’S ABILITY TO MAKE DECISIONS?
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 12
2002: Plowman Promotes A Systems Engineering Process with Decision Making
Existing Technologies
Existing Designs
Existing Facilities
Customer/ User Needs
External Influences
• Product • Process • System • Facility
Know What the Customer Wants and How the System Must Perform
(Mission / Requirements Analysis)
Program Requirements Rules, Orders,
Regulations, Standards Technical
Requirements
Plan and Integrate Your Work
(Systems Engineering Planning, Baseline Management,
Tradeoff Studies System Integration)
Verify the Preferred Solution
(System Verification)
Detailed Specifications
System Boundary SE Checklist
Customer Needs/Expectations List
System Requirements List (with Customer Approval)
Top-level Functional Architecture Alternatives Designs (e.g., sketches, designs, etc.)
Decision Criteria (“System has to ...” list with
measurable units)
Concept Matrix (Decision Criteria vs. Alternatives)
System Engineering Management Plan (SEMP) Document Control Log
Customer Requirements and System Requirements Documents
System Performance Measures Initial Program and Technical Interface Requirements
Formalized Functional Hierarchy Detailed Alternative Concepts
Decision Criteria System Functional Architecture Technical, Cost Baselines
Trade Studies/Analysis Plans and Results
System Requirements Review
System Requirements Baseline
Preliminary Design Review
System Baselines Revisions
Conceptual Design Review
System Baselines Revisions
Verification Matrix
Design Reviews
Systems Engineers at the Idaho National Engineering and Environmental Laboperform the following roles in support of Lockheed Martin Idaho Technologies Company and U.S. Department of Energy programs and projects:
¼ assess specific program and customer needs and expectations through interaction and facilitation of groups meetings and discussions
¼ implement an appropriate graded SE approach to meeting program and customer needs
¼ identify and manage system requirements, including the use of requirements management tools
¼ identify and develop system alternatives
¼ analyze system alternatives to ensure program requirements are met, including evaluating system risk, cost, and other performance measures using computer simulation models and other techniques
¼ provide “what-if” scenario modeling for sensitivity analysis of changes in program requirements and/or system variables
¼ document and validate analysis results
¼ support the decision making process
Test Plans
Systems Engineers at the Idaho National Engineering and Environmental Laboperform the following roles in support of Lockheed Martin Idaho Technologies Company and U.S. Department of Energy programs and projects:
¼ assess specific program and customer needs and expectations through interaction and facilitation of groups meetings and discussions
¼ implement an appropriate graded SE approach to meeting program and customer needs
¼ identify and manage system requirements, including the use of requirements management
tools
¼ identify and develop system alternatives
¼ analyze system alternatives to ensure program requirements are met, including evaluating system risk, cost, and other performance measures using computer simulation models and other techniques
¼ provide “what-if” scenario modeling for sensitivity analysis of changes in program
requirements and/or system variables
¼ document and validate analysis results
¼ support the decision making process
¼ support product development and implementation do
Determine Desired Functionality
to Meet Customer Needs (Functional Analysis)
Functional Hierarchy
Functional System Architecture
Study Various Options and
Determine a Preferred Solution (Alternatives Analyses / System Synthesis)
Analytical Data
Models & Simulations
Parametric Data
Ranked Decision Criteria
Trade Studies
Physical System Architecture
13 Buede, Forsberg, Mooz, Plowman, and Tufts. “Systems Engineering Body of Knowledge, Section II: Systems Engineering Processes”, INCOSE INSIGHT, April 2002
2007: Jain, Squires, Verma, & Chandrasekaran Indicate We Might Have A Problem in SE Education
“The research revealed that the following three core courses had a weak relationship or absence of any relationship with the other topical areas:
– Quality, safety, and systems suitability – Modeling, simulation, and optimization – Decisions, risks and uncertainty
The most serious gaps were noticed between the above three core courses and the three specialized, elective courses below:
– General project management – Finance, economics, and cost estimation – Organizational leadership”
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 14
Jain, Squires, Verma, and Chandrasekaran. A Reference Curriculum for a Graduate Program in Systems Engineering, INCOSE INSIGHT, July 2007
Mission Analysis / Strategic
Planning
Needs Assessment /
Functional Analysis
Concept Selection /
System Architecture
Project Planning
SE / VE Planning
Customer Concept of Operations
Requirements Development
Alternatives Evaluation / Synthesis
Detailed Design / Specifications
Monitor Development / Prepare for IV&V
Subsystem & Component Integration
System Verification
System Validation
Process Improvement
Modification Management
System Retirement or Replacement
Life Cycle Time Line
Phase -‐1
Preconceptual / Mission Analysis
Phase 0
Concept Explora:on, Benefits Analysis, and Project Planning
Phase 1
Tech. Planning / Concept of Opera:ons
Phase 2
System Defini:on, Design, and Development
Phase 3
System Integra:on, Implementa:on, Verifica:on &
Valida:on
Phase 4
Opera:ons, Maintenance, and Modifica:on Management
Phase 5
System Re:rement / Replacement
Dep
loym
ent
Rev. 5, May 6, 2009 INL/MIS-09-16067
Alternatives Development / Trade Studies
CD-0 CD-1 CD-2 CD-3 CD-4
Typi
cal
Pro
ject
Li
fe-c
ycle
Typi
cal
Dec
isio
n P
oint
s Sy
stem
s Eng
inee
ring
Fun
ctio
ns
Subsystem & Component Verification
Cross-Cutting Activities (Alphabetical)
Modeling & Simula:on
Human Systems Integra:on
Configura:on Management
Decision Analysis (e.g., Metrics)
Program / Project Integra:on (e.g., Lead SE)
Stakeholder Analysis / Elicita:on
Risk Management
System Life Cycle Management
Requirements Management
Facilita:on & Consulta:on / VE
Reliability, Availability, Maintainability, Inspectability
Decision Planning (e.g., Roadmapping)
2009: Idaho National Lab Reinvents the Vee-Model to Include Decision Making
15
2009: De Weck Invents a New Vee-Model for His Course Syllabus
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 16
Olivier de Weck, 16.842 Fundamentals of Systems Engineering, Fall 2009. (Massachusetts Institute of Technology: MIT OpenCourseWare), http://ocw.mit.edu (Accessed April 5, 2012). License: Creative Commons BY-NC-SA
2010: GRCSE Survey of Systems-Centric Master’s Programs Shows Decision Analysis as Visible But on the Fringe
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 17
Top$10$Courses*$Identified$within$the$Program!Introduction$to$SE! 56%!Modeling$and$Simulation! 53%!Project$Management! 38%!Systems$Architecture$and$Design! 38%!Systems$Integration! 34%!Systems$Analysis! 25%!Systems$Management! 25%!Systems$Requirements$Analysis! 25%!Risk$and$Decision$Analysis! 25%!Probability$and$Statistical$Analysis! 19%!
!Graduate Reference Curriculum for Systems Engineering (GRCSE), version 0.5, December 2011
2011: INCOSE Decision Analysis Working Group Chartered
INCOSE�Decision�Analysis�Working�Group�(DAWG)�Charter�
3 of 3
9 RESOURCE�REQUIREMENTS��TPP�will�be�submitted�for�specific�projects�as�required.�
10 DURATION�This�Charter�will�remain�in�effect�until�rescinded�by�the�signatory.�����
11 SIGNATURES�
Chair:�Frank Salvatore� � � � � � Date:�2011Ͳ03Ͳ12�
� � � � � � Date� � August�23,�2011�
Technical�Director,�INCOSE��
Chairman,�INCOSE�Board�of�Directors�� � � � Date�
�
Revision�History��
Date� Revision� Description� Author�
2001Ͳ03Ͳ12� 1.0� Initial�Draft.� Frank�Salvatore�
� � � �
� � � �
� � � �
�
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 18
2012: 31 Papers Self-Identified as Relating to Decision Analysis / Management at this Symposium
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 19
0" 5" 10" 15" 20" 25" 30" 35" 40" 45" 50"
SEE09"Transi1on"/"Opera1on"/"Maintenance"
SEE18"Environmental"Compa1bility"
SEE17"Logis1cs"/"Supportability"
SEE12"Cause"Analysis"
SEE16"Reliability"/"Availability/Maintainability"
SEE20"Resource"Management"
SEE21"Safety"/"Security"
SEE02"Complexity"Science"
SEE03"Systems"Dynamics"
SEE22"LifeKcycle"Cos1ng"/"Economic"Evalua1on"
SEE23"Acquisi1on"/"Supply"
SEE19"Human"Factors"/"Human"System"Interface"
SEE15"Measurement"
SEE14"Configura1on"/"Informa1on"Management"
SEE29"Other"SE"Enablers"
SEE08"Verifica1on"/"Valida1on"
SEE04"Systems"Science"
SEE13"Risk"/"Opportunity"Management"
SEE07"System"Integra1on"
SEE25"Systems"of"Systems"(SOS)"
SEE05"Requirements"Elicita1on"and"Management"
SEE26"Modelling"and"Simula1on"
SEE27"Teaching"and"Training"
SEE11"Decision"Analysis"/"Management"
SEE10"Project"Planning"/"Assessment"/"Control"
SEE24"MBSE"
SEE01"Systems"Thinking"
SEE06"Architectural"Design"
SEE28"Processes"
Bergsjö. Papers and Posters for the 2012 International Symposium: A Great Variety of Topics, INCOSE INSIGHT, April 2012
Number 6 on the Pareto chart!
Conclusion • “Industry will always be ahead of academia in systems engineering”* • Industry is sending signals that there is a need for better decision
making • Academia needs to step up their game
– Recognize the need in curriculum and course development – Provide training that is applicable for both Discovery and Program SE
environments • System acquirers and producers need to step up their game
– Recognize that Discovery SE and Program SE both exist – Support better decision-making in their both environments – Incentivize better decision making
22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 20
* Gerard Voland, Former Dean, College of Engineering, Technology, and Computer Science, Indiana-Purdue Fort Wayne, in a talk given at the INCOSE Crossroads of America Mini-Conference, October 2004