Introduction to Risk Assessment in Engineering: With Application to Heat Shield Reliability Modeling

Presented by:

Austin HowardUniversity of IdahoMechanical Engineering Dept.Idaho Space Grant Consortium

2Austin Howard

3Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

4Austin Howard

Purpose of This TalkPurpose of This Talk

Describe importance of risk assessment

Introduction to key tools, processes, and concepts related to risk analysis

Provide context with case study based on experiences at summer internship at NASA Ames 2006

Note: Risk assessment is its own discipline and therefore it is outside the scope of this talk to show you how to create/evaluate risk models

5Austin Howard

Definition: RiskDefinition: Risk

Risk: “The combination of the frequency, or probability, of

occurrence and the consequence of a specified hazardous event” -www.bees.unsw.edu.au/ohs/definitions.html

One of many ways to calculate risk: Risk=(Probability of failure)x(Severity of the Consequence)

6Austin Howard

RiskRisk

Risk is also a board game:

7Austin Howard

Risk vs. UnreliabilityRisk vs. Unreliability

Risk is not the same as Unreliability Reliability: Probability that a device will function without

failure over a specified period of time or amount of usage

Reliability is one of the (but not the only) factors that contributes to system risk

Reliability analysis is often used interchangeably with risk, but they are two different concepts

Engineers often present reliability statistics rather than risk values due to difficulty of measuring and comparing consequence severity

8Austin Howard

Risk vs. SafetyRisk vs. Safety

Judging Risk is a quantitative activity grounded by testing, and physical

modeling

Judging Safety is a qualitative, political activity

You must have a safety standard to judge system risk against otherwise risk is a relatively meaningless value in decision making and design assessment

9Austin Howard

Deterministic vs Non-DeterministicDeterministic vs Non-Deterministic

Deterministic model-model behaves predictably In other words, for a constant input, you will always get

the same output

Non-deterministic model-model with one or more choice points where different continuations are possible In other words for a constant input, you will not always

get the same output Requires input from one or more: user, global variables,

hardware timer, random numbers, stored data…

10Austin Howard

Purpose of Risk AssessmentPurpose of Risk Assessment

Purpose of Risk Assessment: Answering and effectively communicating the following questions/considerations:

Haimes, Yacov Y. Risk Modeling, Assessment, and Management.Hoboken, NJ, USA: John Wiley & Sons, Incorporated, 2005. p 23.http://site.ebrary.com/lib/uidaho/Doc?id=10114200&ppg=47

11Austin Howard

Reputation

Customer Satisfaction/Safety

Warranty Costs

Repeat Business

Cost Analysis

Customer Requirements

Competitive Advantage

Importance of Risk AnalysisImportance of Risk Analysis

12Austin Howard

Cont…Cont…

Reduce long term cost

http://klabs.org/DEI/References/design_guidelines/analysis_series/1314msfc.pdf

13Austin Howard

ProcessProcess

Risk Communication/Safety Check

Component

Sub System

System

Model and/or Test

Sub System Tree(Fault and/or Event)

Risk AssessmentEx. FMEA

System Tree(s)(Fault and/or Event)

Sub System Tree(Fault and/or Event)

Model and/or Test

Model and/or Test

Model and/or Test

Production

PassRisk

Mitigation

Fail

14Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

15Austin Howard

Failure Mode Effect Analysis Failure Mode Effect Analysis (FMEA)(FMEA)

Other wise known as: Failure Mode Effect Criticality Analysis (FMECA) Design Failure Mode Effect Analysis (DFMEA) Process Failure Mode Effect Analysis (PFMEA)

Purpose Define and guide a logical design process Identify, quantify, and reduce design risk Provide a traceable document for design and development Justify design activities Provide a means for continuous product improvement

16Austin Howard

Cont…Cont…

Combines Possible Failure: Severity (rate 1-10) Occurrence (rate 1-10) Detect-ability (rate 1-10)

Product of the parameters is called the RPN, this value describes the overall risk of each failure mechanism

High RPN numbers = high risks Focus on these failure mechanisms first in risk mitigation

process

17Austin Howard

FMEA ProcessFMEA Process

http://www.qualitytrainingportal.com/resources/fmea/fmea_process.htm

18Austin Howard

Example: FMEAExample: FMEA

19Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

20Austin Howard

Fault TreesFault Trees

At the top of a fault tree is a failure

Under the tree are all the possible faults that could lead to the top failure

Fault trees are used for viewing a system and the interactions between faults and possible paths to a failure

Fault trees can be built with software and combined with probabilities to produce reliability estimates

21Austin Howard

Cont…Cont…

Paths from bottom to top of tree are termed cutsets, the shortest cutset is the minimum cutset

Symbols used:

Haimes, Yacov Y. Risk Modeling, Assessment, and Management.Hoboken, NJ, USA: John Wiley & Sons, Incorporated, 2005. p 530.http://site.ebrary.com/lib/uidaho/Doc?id=10114200&ppg=554

22Austin Howard

Example: Fault TreeExample: Fault Tree

http://safety.transportation.org/htmlguides/implement/ProcAppJ.htm

23Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

24Austin Howard

Event TreesEvent Trees

Goal of event tree to determine the probability of an event based on the

outcomes of each event in the chronological sequence of events leading up to it

By analyzing all possible outcomes using event tree analysis, you can determine the percentage of outcomes which lead to the desired result

Event trees can be built with software to produce reliability estimates

25Austin Howard

Example: Event TreesExample: Event Trees

http://www.ece.cmu.edu/~koopman/des_s99/safety_critical/

26Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

27Austin Howard

TestingTesting

AdvantagesCan illuminate overlooked failure mechanismsSome situations cannot be modeled accurately with

current physical understanding Turbulence

LimitationsExpensiveTime consumingNeed lots of data to be meaningful

28Austin Howard

How Modeling Produces How Modeling Produces UnreliabilityUnreliability

Load Probability Curve

Design Probability Curve

Area=Probability of failure

Mean LoadMean Design

SpecDesign Margin

29Austin Howard

ModelingModeling

AdvantagesCan be relatively inexpensive/fast

LimitationsEasy to make incorrect assumptions/mistakesSome situations are difficult/impossible to model

accurately

30Austin Howard

System/Sub-System ReliabilitySystem/Sub-System Reliability

Series Reliability

A B C

Rtot = RA * RB * RC

Full RedundancyA

B

C

Rtot = 1- (1- RA ) * (1 - RB) * (1 - RC)

31Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

32Austin Howard

The Essence of Monte CarloThe Essence of Monte Carlo

Monte Carlo: Method of modeling involving inputs from random or pseudo random numbers

Output produced has the similar characteristics to that of data collected from an experiment*Similar scattering of dataThe more “runs” of the model, the more pronounced

the trends are

*If input is correct - your model output is only as good as the information you put into the model

33Austin Howard

What Monte Carlo Looks LikeWhat Monte Carlo Looks Like

Vose, David; Quantitative Risk Analysis:A guide to Monte Carlo simulation modeling; 1996

34Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

35Austin Howard

Heat Shields 101Heat Shields 101

Kinetic Energy: +Potential Energy: Thermal Energy (hot)

Entry velocities between 7km/s(LEO)-11km/s (Lunar return), Altitude ~400 km (+ for lunar return)

Blunt body advantage Shuttle vs Apollo

mgdy2

2

1mV

36Austin Howard

Cont…Cont…

Apollo Shuttle

Bef

ore

Aft

er

37Austin Howard

Case Study ObjectivesCase Study Objectives

Risk Assessment Objectives For Orion Heat Shield:

Obtain an estimation of the overall system reliability

Identify components/events most likely to cause failure

Identify sub-systems that may be too conservative

Determine sensitivity of design/modeling/testing/environmental parameters on system reliability Determine where resources should be allocated in order to

reduce risk most efficiently

38Austin Howard

Failure ModesFailure Modes

TPS Failure Modes

Burnthrough of heat shield material

Crack

Damage

De-bonding

Hot spots

Flowthrough

Bondline overheat

Excessive conduction

Radiation absorption

System interface failure

e.g. electromagnetic interference, landing system interference

39Austin Howard

The Software UsedThe Software Used

SAFE – Space Architecture Failure EvaluationCode in development at NASA AmesMonte Carlo Simulation method

Input Assembly architecture Nominal reliabilities of components and events Consequences of failure Mission outline (events and segments)

The software generates hundreds or thousands of semi-random repetitions of the given scenario

The output Histograms and mission summaries that engineers can use to

determine when the system is likely to fail, what will cause failure, and how often system failures are likely to occur…

40Austin Howard

Simple ExampleSimple Example

41Austin Howard

Risk Interaction ExampleRisk Interaction Example

Micro-Meteoroid and Orbital Debris (MMOD) Risk of significant sized particles hitting heat

shield with significant velocity to cause damage

Risk of the MMOD damage causing/contributing to TPS failure

42Austin Howard

Another ExampleAnother Example

Environment modeling Accurately predicting entry environment

Recession modeling based on predicted environment

Material selection/Thickness design based on recession modeling

43Austin Howard

Organizing the RisksOrganizing the Risks

44Austin Howard

Visualizing Risk InteractionVisualizing Risk Interaction

45Austin Howard

Calculating Risk ValuesCalculating Risk Values

46Austin Howard

The ModelThe Model

47Austin Howard

Predicting ReliabilityPredicting Reliability

Historical records Apollo Shuttle Others

Physics based simulation tools

Testing Ground Tests Flight Tests

48Austin Howard

Results of Summer WorkResults of Summer Work

Reliability model: Incorporates over 90 potential TPS risks

Each risk can fail in either a benign or catastrophic manner

Multiple benign failures have the ability to contribute to a catastrophic failure

All pre-entry factors influence risks during entry and landing phases

Risk Analysis Document Outline for detailed sub-system interaction Can be used to track changes and understand model Can be used to help understand risk dependence on material

choice and other design factors

49Austin Howard

OutlineOutline

Introduction Failure Mode Effect Analysis (DFMEA) Fault Trees Event Trees Obtaining Component Reliability Monte Carlo Method Case Study: Heat Shield Reliability Modeling Summary

50Austin Howard

SummarySummary

Risk analysis is a large topic that describes an entire discipline of engineering

Risk analysis is an iterative process If used correctly, can save money, and lives!Can aid in decision making process, justify actions

There are lots of tools available for engineers

51Austin Howard

Cont…Cont…

The output of a risk assessment is only as good as the input The engineer must have plenty of test data or a sound

model before a valid risk model can be produced

Model output is meaningless without bounds on the solution

52Austin Howard

Questions?Questions?