Shared Knowledge and Information Flow in Systems Engineering Socio-Cognitive Analysis of the GSFC...

Shared Knowledge and Information Flow in Systems EngineeringSocio-Cognitive Analysis of the GSFC Mission Design Laboratory

Mark S. AvnetPh.D. CandidateEngineering Systems DivisionMassachusetts Institute of Technology

NASA Goddard Space Flight CenterSystems Engineering SeminarMarch 3, 2009

Mark S. AvnetGSFC Systems Engineering Seminar

March 3, 2009Slide 2 of 37

• S.B. in Physics, MIT, 2001; M.A. in Space Policy, GWU, 2005• Software Engineer, 2001 – 2003; NASA HQ, 2004 – 2005

• Observed an Interesting Phenomenon― Decisions in space systems development require

integration of perspectives: policy, scientific, engineering, public, etc.

― Systems engineering takes into account the unique views of each, but the engineer is taken to be outside of the stakeholder framework.

• Ph.D. in Engineering Systems, MIT, 2009― Research addressing this issue― Focus of this talk: contributions to SE here at GSFC

Who Am I and Why Am I Here?Who Am I and Why Am I Here?



Source: Robinson, G.L., “Systems Engineering Initiatives at NASA,” Goddard/SMA-D Education Series, 25 Sept 2008.

Perspectives on Space Systems DesignPerspectives on Space Systems Design



Overview of the Mission Design Lab

Analysis of the Design Process

A Model of Shared Knowledge

Integrated Analysis: People and Process

Structure of the PresentationStructure of the Presentation



Overview of the Mission Design Lab

Part 1



Integrated Design Center (IDC)

Mission Design Lab (MDL)

Instrument Design Lab (IDL)

Focus of this Talk

GSFC Integrated Design CenterGSFC Integrated Design Center



The Mission Design LabThe Mission Design Lab



http://idc.nasa.gov/mdl/products.cfm

http://idc.nasa.gov/idc/services.cfm

The MDL: Structure and ProductsThe MDL: Structure and Products



The MDL: Roles and FacilityThe MDL: Roles and Facility



MDL Design Study ObservationsMDL Design Study Observations

.

“Typical” Studies



Analysis of the Design Process

Part 2



Task A depends on information from Task G

Tasks D and E must be done concurrently

The Design Structure Matrix (DSM)The Design Structure Matrix (DSM)



Series

Coupled

Parallel

Coupled

Phases of the Design Life Cycle

Starting Assumptions

Design Process AnalysisDesign Process Analysis

Series



Modeling the MDL Design ProcessModeling the MDL Design Process



Requirements Definition Phase

EngineeringDesign Phase

Integration Phase

Maintenance andSupport Phase

Costing Phase

Partitioning the DSM: Partitioning the DSM: The Conceptual Design LifecycleThe Conceptual Design Lifecycle



Spacecraft Bus Loop

Propulsion Sizing Loop

Stabilization Loop

Ground Segment Loop

Data Loop

Power System Electronics Loop

Power Loop

Electrical Heating Loop

Propulsion Thermal Control Loop

Radiator Operation Loop

Reentry Loop

Computing Reliability Loop

Radiation Shielding Loop

13 Core Loop Types

Critical Design Trades and Critical Design Trades and Interdependent DisciplinesInterdependent Disciplines



Tear the Design Budgets

Power Budget

Mass Budget

Reliability Budget

Tearing the DSM: Tearing the DSM: Indentification of Starting AssumptionsIndentification of Starting Assumptions



Requirements and Assumptions Phase

Sequential EngineeringDesign Phases

Integration Phase

Costing Phase

Orbit Determination Phase

Itera

teIte

rate

The Torn DSM: MDL Process with The Torn DSM: MDL Process with Starting Assumptions MadeStarting Assumptions Made



Avionics

Communications

Electrical Power

Flight Dynamics

Mechanical

Mission Operations

Thermal

Data Loop Ground Segment Loop

The Core of Interdependent DisciplinesThe Core of Interdependent Disciplines



The Design Structure Matrix is a powerful tool for describing and analyzing the space systems design process.

(Results for your system may vary.)

Insights from DSM-Based AnalysisInsights from DSM-Based Analysis



A Model of Shared Knowledge

Part 3



“Mechanisms whereby humans are able to generate descriptions of system purpose and form, explanations of system functioning and observed system states, and predictions of future system states”*

Mental Models

Condition in which two people utilize the same underlying mechanisms or at least utilize mechanisms that lead to similar descriptions, explanations, and predictions

Shared Mental Model (SMM)

Team Member

Team Member

SMM

* Rouse, W.B. and N.M. Morris (1986). “On Looking Into the Black Box: Prospects and Limits in the Search for Mental Models.” Psychological Bulletin 100(3): 349–363.

Mental Models of the SystemMental Models of the System



Measuring Mental ModelsMeasuring Mental Models

Survey Question on Major Design Drivers



Dx = # of drivers selected by person xDy = # of drivers selected by person yDx,y = # of drivers selected by both x and y

yx

yxyx DD

DS ,

, 2

Mental Model Sharedness, Sx,y , is defined as:

Measuring Shared Mental ModelsMeasuring Shared Mental Models

Team Member x

Team Member y

Sx,y

Ratio of common choices to total choices



Social Network AnalysisSocial Network Analysis

A set of tools and techniques for analyzing a large group of entities (nodes) and the structure of interactions and/or relationships among them (edges).

Node

Edge

Node = Design Team Member x or y

Edge = Shared Mental Model between x and y

Edgeweight = Value of Sharedness, Sx,y



Pre-Session Post-Session

CSMM = structural similarity (edge-by-edge correlation)

2

1 SMMCS

Dynamics of Shared KnowledgeDynamics of Shared Knowledge

Change in Shared Knowledge



Dynamics of Shared Knowledge: Dynamics of Shared Knowledge: Relationship to System AttributesRelationship to System Attributes



Integrated Analysis: People and Process

Part 4



IP,Comm = proportion of team checking Communications

Content of Shared Knowledge: Content of Shared Knowledge: Perceived Importance of DriversPerceived Importance of Drivers



Recall the Central Role of Communications in the Design Process

The Communications Subsystem: The Communications Subsystem: An Indicator of Shared KnowledgeAn Indicator of Shared Knowledge



Actual Interaction Matrix

Based on Survey Data of Interactions for Each Study(Study 3 Shown Here)

Expected Interaction Matrix

Based on Core Loop Types in the Partitioned DSM

Measuring Team CoordinationMeasuring Team Coordination



N

NNC b

TS

#

Congruence Matrix

Overlay of Expected and Actual Interactions

Socio-Technical CongruenceSocio-Technical Congruence

N# = number of # cells

Nb = number of blank cells

N = total number of cells



Dynamics of Shared Knowledge: Dynamics of Shared Knowledge: Relationship to Team CoordinationRelationship to Team Coordination



Period of learning and consensus building

Sub-teams based on interdependent disciplines

Determine starting assumptions

Resolve orbit determination trades

Design sequentially… then iterate

DSM-based process automation software

Lab layout based on interdependent disciplines

People Process Tools

Facility

The Typical MDL Process: The Typical MDL Process: Recommendations in DiscussionRecommendations in Discussion

Proposed Standard Design Process Model under Development in Conjunction with the MDL



The People Behind This WorkThe People Behind This WorkAnnalisa Weigel, MIT, Thesis AdvisorNASA Graduate Student Researchers Program (GSRP)

Deborah Amato, Former IDC Systems EngineerJennifer Bracken, IDC Systems EngineerTammy Brown, IDL Team LeadBruce Campbell, IDC ManagerAnel Flores, MDL Systems EngineerGabriel Karpati, Former IDC Systems EngineerJohn Martin, MDL Team LeadMark Steiner, SESAC Branch Head

IDC Support Staff: Felicia Buchanan-Jones, Dawn Daelemans, Elfrieda Harris, Erica Robinson, Ed Young

And, of course, the MDL engineers, whose sustained participation made this work possible.

12 MDL Customer Teams



Thank You



Backup



Building the DSM for the MDLBuilding the DSM for the MDL

• Steps of DSM Construction in the MDL1) Preliminary Interviews2) Surveys on Design Sessions3) Structured Interviews4) Verification and Validation

• Parameter-Based DSM

• Guiding Principles for DSM Construction in the MDL– Document maximal flow for a typical design session– Include only deliberate and purposeful information flow– Abstract two-way negotiation-type interactions

Although collocation accelerates the pace of design activity, it also presents an obstacle to formal analysis and process improvement. DSM construction must account for this.



Data Collection on Mental ModelsData Collection on Mental Models

24 = 16 Possible Mental Models

• Survey Data on Major Design Drivers― Team members indicate whether each of a set of issues

drives the ultimate design.• Simple Example with Only Four Possible Drivers

― Cost― Schedule― Performance― Science



Filtering Out Random Responses: A Filtering Out Random Responses: A Cutoff For Shared Mental ModelsCutoff For Shared Mental Models

x and y do not share mental models to any greater extent than two people with no prior knowledge of the task answering at random

SMMx,y = 0 SMMx,y ≥ 1

35 Possible SMMs

EVS yx ,



• Surveys Distributed: 20 Drivers and 1,771 Possible SMMs• Network Edge Weights on a 1-4 Scale

• Time Dependence of Shared Knowledge― 12 Design Sessions Observed― Pre- and Post-Session Data Collected for Each

Quantifying Shared Knowledge: Quantifying Shared Knowledge: Edge Weights in a Social NetworkEdge Weights in a Social Network

:4

:3

:2

:1

:0

,

,

,

,

,

yx

yx

yx

yx

yx

SMM

SMM

SMM

SMM

SMM

5.1

5.1

5.0

5.0

,

,

,

,

,

EVS

EVSEV

EVSEV

EVSEV

EVS

yx

yx

yx

yx

yx

.



Dynamics of Shared Knowledge: Dynamics of Shared Knowledge: Relationship to System AttributesRelationship to System Attributes



Propulsion Subsystem and Mission TypePropulsion Subsystem and Mission Type



Team Coordination and Shared Team Coordination and Shared Knowledge in the TeamKnowledge in the Team



Proposed Standard Design Proposed Standard Design Process ModelProcess Model



• Product Development― Guiding Principles for Building a Design Structure Matrix in

a Rapid Collaborative Design Environment― Method for Converting a Parameter- to a Team-Based DSM

• Cross-Functional Teams and Shared Mental Models― Scalable Network Model of Shared Knowledge in

Engineering Design― Metric that Captures Dynamics of Shared Knowledge

• Systems Engineering and Space Systems Design― System-Level Representation of the Entire Design Process― Analysis of the Role of People in the Process― Standardized Design Process Based on Both of the Above

• Explicit Connection between Organizational/Social Psychology and Systems Engineering Best Practices

Contributions to the ResearchContributions to the Research



1) Apply Methods to the Instrument Design Laboratory and to Other Similar Design Centers; Apply Both DSM and SMM Work to Longer Development Programs

2) Build DSM with Types and Strengths of Dependencies3) Time Series Analysis – 1 to 2 Surveys Each Day Tracking

the Evolution of SMMs Over Time4) Measure SMMs Based on Other Forms of Knowledge in

Addition to Task – Team, Process, Context, Competence5) Network Analysis of Design Sessions6) Experimental Approach with a Learning Period Structured

in Various Ways and Several Combinations of Number and Length of Design Iterations

Future WorkFuture Work

Shared Knowledge and Information Flow in Systems Engineering Socio-Cognitive Analysis of the GSFC...

Documents

Transcript of Shared Knowledge and Information Flow in Systems Engineering Socio-Cognitive Analysis of the GSFC...