I wouldn't do that if I were you - Critical Uncertainties...know there are some things we do not...

M AT T H E W S Q U A I R

I W O U L D N ’ T D O T H AT I FI W E R E YO U

N O N E S U C H P U B L I S H I N G

Copyright © 2017 Matthew Squair

published by nonesuch publishing

First printing, August 2017

Introduction

A nuclear reactor’s defences are overwhelmed by a tsunami, an aircraft manufacturer’s newest aircraft is broughtlow by design faults while another aircraft’s crew is overwhelmed by a cascading series of failures. Why do thesethings happen? Can we predict such events and protect ourselves from them, or are we destined to suffer likedisasters again and again?

A very bad fire

Figure 1: The Apollo 1 capsule before thefire (NASA Image)

it’s an ordinary day at Cape Canaveral, the launch date forApollo 1 is a month away and in preparation a ‘plugs out’ test ofthe Apollo command module underway. As the name implies thistest requires onboard systems to be operated stand alone, so thecrew are embarked. The module has also been sealed to to conductconcurrent pressurisation tests which requires the pressure insideto be raised above atmospheric in order to properly seal the plugdoor. All appears to be going smoothly but then suddenly over thecrackling intercom comes a cry from the capsule

‘Fire!, ... got a fire in the cockpit!’

Before launch pad personnel can react a sheet of flame eruptsfrom the Apollo module into the clean room. Over the CCTVhorrified personnel can see through the hatch window silver armsfumbling with the hatch before the room fills with toxic smokeWithin minutes three astronauts are dead.1 1 Courtney Brooks, James M. Grimwood,

and Loyd S. Swenson Jr. Chariots for Apollo:A History of Manned Lunar Spacecraft.NASA History Series. NASA, 1976

Figure 2: The interior of the Apollo 1capsule after the fire (NASA Image)

The investigation

So how and why did this tragedy happen? According to the formalaccident investigation, the accident occurred because of a combina-tion of the following major factors2:

2

• an ignition source most probably “vulnerable wiring carry-ing spacecraft power” and “vulnerable plumbing carrying anethylne glycol combustible and corrosive coolant”.

• a build up of “many types and varieties” of flammable materialssuch as paper documentation, velcro and nylon webbing withinthe capsule.

12

• a pressure test that induced greater than ambient and pureoxygen3 atmosphere which turned what would have a localisedignition into a raging inferno.

3 An increase in oxygen levels can turn ma-terials that are considered non-flammable(such as Nomex cloth) under normalatmospheric conditions into flammableones.

• the inability of the crew to quickly escape due to the plug stylehatch design which, given the greater than atmosphere internalcapsule pressure, meant the crew could not open the hatchquickly.

• a failure to recognise the risk associated with the pressure test sothat the operational planning and preparation was inadequate,including for emergency response.4

4 Failing to comprehend the risk led in turnto the overlapping of two separate testactivities the plug out test of the vehicle andthe pressure test.

These factors in combination led with remorseless logic to the oc-currence of a severe fire in the capsule that overwhelmed the crewbefore they were able to open the hatch and make their escape. Theinvestigation board made numerous technical recommendationsthat in turn drove significant changes to design, manufacturing andprocedures. But that’s not quite the whole story.

Culture and complacency

In the days after the accident previously overlooked symptomsof a deeper malaise in the National Aeronautics and Space Ad-ministrations (NASAs)’s safety efforts started to emerge. Ratherthan a seamless sequence of near perfect triumphs it turns out thatNASA’s missions had in fact been dogged by a series of close calls;from the failure of John Glenn’s retro-pack, to Eugene Cernan’sdisastrous first spacewalk or Gus Grissom’s near drowning. Indeedamongst these incidents there had been several involving fires inoxygen enriched atmospheres. In one 1962 incident during mannedtests of the Gemini space suits in a pure oxygen chamber a firebroke out destroying the chamber, luckily without injury to thetest subjects. With that perfect clarity that hindsight brings NASAnow appeared as an organisation lulled into complacency by it’ssuccesses, and unable to perceive the harbingers of disaster for theApollo program. A contemporary cartoon by Paul Conrad 5 sums 5

up this view, counterpointing the feeling of disbelief within theprogram that after the successes of Mercury and Gemini that suchan accident could occur.

The ensuing US senate enquiry and it’s publicity blew like a coldwind through both NASA and NAA removing or sidelining manywho were deemed to have been responsible for the disaster. Aspart of the organisational reform of NASA the Office of Safety andMission Assurance (OSMA) was established under the leadershipof Jerry Lederer, then Director of the Flight Safety Foundation(FSF) to address what NASA upper management saw as an im-balance between the program objective of placing a man on themoon within the decade and crew safety. As Jerry Lederer put itsuccinctly, “...You always have to fight complacency - you need

13

formal programs to ensure that safety is always kept in mind”.

Choices, contracts and constraints

But the reasons for the fire go further than simple manufacturingdefects or managerial malfeasance but to fundamental designchoices, and their context, made in the early years of the program.Both mercury and Gemini programs had elected to utilise pureoxygen atmosphere mixes, primarily to save weight as both wereutilising re-purposed military launchers, only with the purposebuilt Saturn launchers did an opportunity for NASA present itselfto move to a safer atmosphere mix. And so it was that the initialrequest for proposal for Apollo released by NASA premised anitrogen-oxygen mix atmosphere. However by 1961 it had becomeapparent that this approach also carried with it significant penaltiesin terms of weight, cost, time and complexity. Given that by 1962

NASA had launched a series of successful Mercury and Geminimissions using a pure gas atmosphere there seemed no good reasonto complicate the Apollo design and by August of 1962 RobertGilruth, Director of NASA’s Manned Spacecraft Centre, had madeoffical the reversion to the Mercury/Gemini design concept througha formal contract change notice.

A mixed gas system would have requireddual tankage along with a sophisticatedmixing system, where in contrast a pureoxygen system would require only a simpletank and pressure gauge. The capsulewould also have needed to maintain ahigher pressure with mixed gas requiringa stronger pressure hull again increasingmass. The use of mixed gases only duringthe high cabin pressure launch phase wasalso proposed by NAA but still carried withit the risk of decompression sickness aswell as the potential for asphyxiation by anitrogen imbalance. In fact seven yearsbefore there had been just such a seriousaccident during tests of a mixed gas systemon the Gemini program.

NASA was in truth well aware of the hazards posed by a pureoxygen environment. There had been previously been a vigorousdebate between NASA and North American Aircraft (NAA), theApollo contractor, as to whether a nitrogen-oxygen mix or a pureoxygen atmosphere was safer. In the end NASA had decided that,fire risk notwithstanding, a pure oxygen based system was safer,simpler and lighter. Fire safety, it was assumed, could be assuredby managing the inventory of flammable materials 6. What was 6

not fully understood by NASA and NAA was that managing thefire risk was not simply an exercise in materials engineering andinventory control. To understand the actual potential for a severefire the total ensemble of materials needed to be considered inrelationship to each other and to the operational environment. Inother words fire safety was a system property. So what of the hatch?Why would NASA and NAA come up with an apparently deficientdesign? The answer again goes back to design decisions madeearly in the life of of the Apollo spacecraft. After the inadvertentactuation of Liberty Bell 7’s pyro driven emergency hatch releaseNASA decided to eliminate the potential for such catastrophicevent by reverting to a manually operated, and therefore muchslower to open, hatch. Unfortunately this also increased the risk ofcrew entrapment in the case of a rapidly spreading fire.

Had the engineers on Apollo been polled as to the risks involved inthe Apollo design prior to the incident the answer they would havegiven would undoubtedly be different to that given if the questionwas asked after the accident. Likewise the causes of the accident

14

seem to be as much rooted in organisational responses to other,apparently more salient risks, rather than simple neglect. NASAwas not oblivious to the risks posed by oxygen fed fires, and in thecontext of that and other risk had made a series of decisions thattaken individually seemed reasonable. Yet in combination theyresulted in the loss of Apollo 1 and the death of three astronauts.Managing risk in an environment of rapid technological change is,it seemed, not as simple an endeavour as NASA’s engineers andand management might have believed.

Why this book

In the short term NASA learned the lessons of Apollo 1, going onto carry out the remaining Apollo missions without further loss oflife.7 However there remains a broader question raised by incidentslike the Apollo 1 fire, and that is, in the face of the uncertaintiesposed by complex systems and new technologies, can we alsoachieve an acceptable level of safety. Answering this question bothphilosophically and practically is the purpose of this book. 7 Although the subsequent Apollo missions

were not without their fair share of incidents.See for example the infamous Apollo 11LEM 1202 software error, the lightningstrike on Apollo 12 during launch or the fuelcell explosion of Apollo 13.

1 What is Risk?

“...there are known knowns; there are things we know we know. We also know there are known unknowns ...weknow there are some things we do not know. But there are also unknown unknowns, the ones we don’t know wedon’t know.”

Donald Rumsfeld

“I often say that when you can measure what you are speaking about, and express it in numbers, you know some-thing about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of ameager and unsatisfactory kind.”

Lord Kelvin

“Oh well, if you can’t measure, measure it anyway”

Frank Knight

1.1 We live in a time of uncertainty

The steady growth in human knowledge regularly produces sud-den and disruptive step changes in human capabilities1. Unfor- 1 For example the steady increase in

transportation speeds over the last 200years led to a ‘sudden’ emergent abilityof the United States to go to the moonwithin a decade of declaring that intent.Likewise Moore’s law leads inevitably to theexplosivee growth of the internet of things.The mathematician Samuel Arbesman callsthese events ‘fact phase changes’.

tunately the dark side of these advances in human potential isfor unanticipated accidents of truly catastrophic magnitude. Theproblem that society faces is that eliminating the possibility of suchaccidents, or mitigating their consequences, seemed to be an impos-sible task. So how to justify the acceptability of such technologiesand their inherent catastrophic potential when experience was nota reliable indicator? The answer found was to take a risk basedapproach. If the probability2of an accident could be shown to be 2 The original use of the word (and retained

in the legal usage of the term) probabilitypredates it’s association with the the ideaof ‘chance’ or ‘randomness’, that is afoundational concept in classical risk theory.For a philosopher’s view of how the conceptof probability has evolved to encompassit’s current meaning see Ian HackingsEmergence of Probability

Ian Hacking. The Emergence of Probability:A Philosophical Study of Early IdeasAbout Probability, Induction and StatisticalInference. Cambridge University Press,2006

extremely low, so the thinking went, then so to was the risk andtherefore we could ignore the potential consequences. Of course ifyou‘re going to use risk to make such decisions, then you also needto describe and quantify it in a credible fashion, thereby placing themathematics of probability and risk right at the heart of managingthe safety of these new technologies.

In theory such quantitative approaches are fairly straight forward.We determine the set of possible accident events and their severities,calculate each accidents probability then multiply it with the numer-ical severity (such as cost) to obtain a quantitative value of risk. Wecan then sum the set of risks to determine the total quantitative risk.Although the mathematical certainty of such assessments mightappear to satisfy Lord Kelvin’s dictum, just because something is

16 i wouldn’t do that if i were you

true in the model does not necessarily make it true in real life. We arein fact assuming that one can use quantitative risk to reason aboutwhat are usually extremely unlikely events, and as we’ll find that’san assumption hedged about with caveats. In this chapter we’lldiscuss how we arrived at the current state of practice, and sketchout the limitations of such risk practices in the real world.

1.2 A classical beginning

“Doubt is unpleasant, but certainty is absurd.”

Voltaire

The formal mathematical treatment of random systems is gener-ally agreed to have started with an exchange of letters betweenPierre Fermat and Blaise Pascal in 1654 on how to fairly divideup the points from an interrupted game of chance. To solve thisproblem Fermet came up with the idea of an ensemble, that is a setof hypothetical parallel worlds that captured all the possible futureoutcomes, and coined the term ‘expectation’ to describe the averagevalue of this ensemble3. Fermet‘s insight was that expectation also 3 Hence the term ‘ensemble average’ which

is the (arithmetic) mean of the ensemble ofpossible states.

represented the fairest way to apportion the winnings. Of coursethis definition is also quietly agnostic as to the consequences oflosing, which is fine as long as the consequences of these losses arenegligible. Before long mathematicians realised that the concept ofexpectation could also be used for other more practical purposes,such as setting insurance premiums or defining pension annuities.Eventually through the work of de’Moivre on shipping insurancethe expectation of losses became the mathematical definition of risk.4

4 A. Hald, A. de Moivre, and B. McClintock.’de mensura sortis’ or ’on the measurementof chance’. International Statistical Review /Revue Internationale de Statistique, 52(3):229–262, 1986

R = S× P (1.1)

From de’Moivre’s definition we obtain Equation 1.1, fairly ob-viously for a given expectation of loss (or risk value R) there’s aresulting inverse relation between severity (S) and probability (P).Plotting these curves gives a series of iso-risk contours Figure 1.2with risk increasing as we move to the right and up in a cartesianfashion. We can utilise this risk value in cost-benefit analyses suchas the example below to decide whether to control a risk or mitigateit in some fashion by comparing the cost of control to that that ofthe expected loss or risk.

1.2.1 Risk as exposure to uncertainty

What is implied, but not stated in the above relationship is thatthere is an exposure to a proposition about which we care and aboutwhich uncertainty exists. If there is complete certainty then risk doesnot exist. For risk to be meaningful it also implies that we mustvalue a specific outcome of the proposition. For example if we are

what is risk? 17

Proposition: Value

Certainty of Loss(No risk)

False(0)

Uncertainty(Risk)

ProbabilityCardinal scale

(0 < p < 1)

Certainty of Gain(No risk)

True(1)

Figure 1.1: Risk as a measure of exposureto an uncertain value proposition.

indifferent to injuries at the level of a paper cut then there is no riskbecause we are indifferent to the outcome and are not thereforeexposed to a potential loss of value that we care about. Risk maybe thought of as exposure to a value proposition of which we areuncertain. In turn we may express our uncertainty as a degree ofbelief utilising the numbers of quantitative pascalian probability.

10−3 10−2 10−1 100 101 102 103 104

10−3

10−1

101

103

105

1

1

10

10

10

100

100

100

Loss severity (S)

Prob

abili

tyof

loss

(P)

de’Moivre’s risk continuum Figure 1.2: Risk contours plotted forvarying risk values on log-log axis. Movingalong a contour ‘risk’ remains the samebut as we move up and to the right eachsucceeding iso-risk contour carries agreater level of risk. These contours providethe construction rules for the risk matricesmuch beloved in risk management.

4 After de’Moivre’s definition, “The Riskof losing any sum is the reverse of Ex-pectation; and the true measure of it is,the product of the Sum adventured (S)multiplied by the Probability (P) of the Loss’.

The formulation of risk as a quantitative calculable measure in turnpaved the way for its treatment as a tractable numerical quantity,whose parameters could be evaluated via pascalian probability.Early quantitative risk assessments such as the WASH1400 (1975)Canvey Island (1978) and Risjnmond (1982) safety studies laid thefoundations of applying classical risk theories in the nuclear andpetro-chemical industries while in aviation the FAA established aquantitative safety target regime for commercial aircraft5. Today, 5 As an example the FAA places an upper

limit on the average probability per flighthours for catastrophic failure conditions of10−9 failures per flight hour.

Federal Aviation Agency. FAA AdvisoryCircular: System Design and Analysis.Advisory Circular AC 25.1309-1, FederalAviation Agency, 1982

even a cursory examination of the state of practice across industrieswhere there is the potential for catastrophic accidents shows usthat quantitative risk is alive and well. Methodologies such asFault Tree Analysis (FTA) and Probabilistic Risk Assessment (PRA)allow risk practitioners to build complex models of risk, standardssuch as IEC 61508, EN 51029 and ISO 31000 normalised the use of


classical risk in decision making about safety while many industryregulators require the quantification of risk in some form as partof their permit to operate process. But underlying all this is thefundamental assumption that ‘expectation’ is the true measure ofrisk, that it is appropriate to use this as a basis for decision makingand that the route to acceptable safety is through the reduction ofprobability.

1.3 An operational definition of classical risk

“In general, we mean by any concept nothing more than a set ofoperations: The concept is synonymous with the corresponding set ofoperations.”

P. Bridgman The Logic of Modern Physics 1927

de’Moivre defined risk as the probability of some loss multipliedby it’s magnitude. The problem is that because risk is a prospectiveattribute, we have nothing that we can directly measure. Howeverwe can in principle identify our exposure to uncertain propositions,characterise their uncertainty using probability and severity inorder to derive the risk value. This is an operationalism approachin that (classical) risk is a priori defined by the set of operations bywhich we use to characterise the probability and severity of someloss. In principle this operational definition saves from the problemof measuring a parameter that cannot be measured directly, thetechnique that we use defines the parameter. In practice howeverwe still have problems with such a defintion, as the followingexample illustrates.

Figure 1.3: The Space Shuttle speed brake.Image source: NASA

case study: the shuttle speed-brake. Actuator gears for theshuttle’s speed brake were installed in reverse on at least two shuttlesand in a high speed landing they would have likely failed with a run-way overshoot and loss of the shuttle as a result. The Rudder/SpeedBrake Actuator was designed for a life of 100 missions over a periodof 10 years with no plans for re-lubrication, maintenance or refurbish-ment. While a reliability study of the actuator was performed by themanufacturer the analysis assumed correct assembly of the gearboxand therefore neglected the potential gear reversal risk6. 6 Fred B Oswald, Michael Savage, and

Erwin V Zaretsky. Space Shuttle Rud-der/Speed Brake Actuator—A Case Study.Probabilistic Fatigue Life and ReliabilityAnalysis. Tribology Transactions, 58(1):186–196, 2014

In this particular case the assumption made hides an uncertainvalue proposition, that the gearbox was installed correctly, byassuming that it was definitely installed correctly. There can bydefinition be no uncertainty, as long as the assumption is accepted.If there is uncertainty over the proposition we would of course careabout it due to the catastrophic consequences and consider it a risk.We should however caveat the prior statement by acknowledgingthat we would care if we actually knew about the risk. As we did notknow at the time of course and therefore could not care this is ofcourse a hypothetical and unobservable state.

what is risk? 19

1.3.1 Operationalising risk perception

The shuttle speed-brake case study illustrates that there is unfor-tunately still a flaw in the operationalism of classical risk. We canin fact be exposed to uncertainty in value propositions (risk) thatwe are unaware of, as the case study of the shuttle’s speed brakesillustrates. Given the assumption of correct installation the beliefis that we need not consider the proposition that it is not correctlyinstalled, so we cannot state that we ‘do care’. Likewise when wecalculate a probability associated with that exposure there can alsobe uncertainties in our calculations that we are unaware of. Ouroperational definition of classical risk cannot unfortunately expressan opinion about uncertainties that we are unaware of.

uncertainty over probability. As an another example I mightpropose a wager on the outcome of a toss of a dice being one. Youmight calculate this as a one in six probability assuming the dice wasfair. However I was actually holding a gaming dice of ten sides. Thusyour estimate of probability contained uncertainties that you werecompletely unaware of.

Operationalism of risk in the classical form as conceptualisedby Fermet and de’Moivre is practically speaking a quantitativemathematical operationalisation of our perception of some risk.Putting it another way when we talk about classical risk, we arereally talking about the operational process we use to articulate ourperception of that probability and loss of a set of identified valuepropositions.

1.4 Quantifying risk?

“To a first approximation, we can say that accidents are almostalways the result of incorrect estimates of the likelihood of one ormore things.”

C. Michael Holloway

Is it possible for us to actually quantify the severity and probabilityof a classically defined risk with sufficient accuracy that our assess-ment is valid? There are good reasons to think that this is not aseasy as we might believe, both in terms of estimating severity, andprobability.7 7 Nasim Taleb calls the belief that one can

characterise real world uncertainties usingthe methods derived from games of chancethe ‘Ludic fallacy’ .

Nasim Taleb. The Black Swan. The Impactof the Highly Improbable. Random House,2007

1.4.1 How we express risk matters

To quantify risk we obviously need an objective way in which wemay express our measurements of the severity and probability. Yeteven the apparently innocuous task of choosing a risk measure fora well-defined loss, such as human fatality, turns out to containcomplex subjective value judgements. For example one mightchoose to express a risk as a reduction in life expectancy, but this


values the young as more important than the old, on the otherhand one might express severity as a simply count of fatalities,but this does not value the voluntary or involuntary nature ofexposure to the risk. Similar questions of judgement arise forprobability. For example we may express it as the probability ofan event over the life of a single system or facility, but this doesnot consider the cumulative probability of an aggregate of likesystems. Alternatively we could express it over some operationalcycle such as trips, distance travelled or operating cycles, but thesemay not reflect the risk exposure of an individual using the systemover some period of time. Decisions as to the basis of measuresfor severity and probability inherently involve subjective, valuejudgements regarding the significance of loss and exposure.

1.4.2 Probability, objective or subjective?

“Though there be no such thing as Chance in the world; our igno-rance of the real cause of any event has the same influence on theunderstanding, and begets a like species of belief or opinion”

Hume, 1748 Enquiry Concerning Human Understanding

Likelihood is expressed numerically using probability, a stan-dard formalism for uncertainty. There are two broad schools ofthought on what probability as concept represents, the Subjectivist(Bayesian) and the Objectivist (Frequentist). The roots of the sub-jectivist viewpoint can be traced back to the work of Hume onempirical knowledge, in a nutshell Subjectivists believe that proba-bility is based on the degree of belief of an observer given all the factsin their possession, the Subjectivist position was perhaps best statedby Finnetti in 1970.8 8 These beliefs still need to be coherent, i.e.

no ‘Dutch books’ and comply with the rulesof probability calculation.My thesis, paradoxically, and a little provocatively, but nonetheless

genuinely, is simply this: PROBABILITY DOES NOT EXIST. Theabandonment of superstitious beliefs about the existence of Phlo-giston, the Cosmic Ether, Absolute Space and Time,...or Fairies andWitches, was an essential step along the road to scientific thinking.Probability, too, if regarded as something endowed with some kindof objective existence, is no less a misleading misconception, anillusory attempt to exteriorise or materialise our true probabilisticbeliefs.

Finnetti (1974 translation, vol. 1, p.x)

Objectivists on the other hand believe that to be a meaningfulconcept probability must be based strictly on observed event sets.The modern objectivist view point can be traced to the work ofPearson, Fisher and Neyman in the 1920s. These champions ofthe objective interpretation developed statistical techniques for theanalysis of variation, sampling techniques, confidence intervals andhypothesis testing. All of which are extremely useful when you aredesigning experiments or analysing large sets of data, which is oneof the reasons that reliability analyses typically involve the use ofthese techniques.

what is risk? 21

If we consider that risk is defined in an operational sense thenwe can treat subjective and objective as simply different opera-tionalisms of probability. The question is not whether one bettercaptures some abstract ‘risk’, but which is more effective for a par-ticular set of circumstances. However the problem we face withtrying to predict the probability of catastrophic or extreme eventsis that data is usually difficult to come by and highly uncertain,driving us towards the subjective judgement of probability andrisk. Subjectivist, or bayesian, probability, as we noted previously,characterises an observer’s background and subjective degree ofknowledge and uncertainty. Therefore when we commit to makingsubjective probability judgements the assumptions, uncertaintiesand potential biases of the assessor should be carefully evaluatedand stated along with the assessed probability. Unfortunately thatusually does not happen, nor are experts particularly circumspectin making such judgements, nor cautious in believing their per-formance within their domain of expertise to be better that that ofnon-experts.

Kahneman et al reviewed a series of studies in which expertswere asked to estimate the highest and lowest future values forprocesses with which they were familiar.9 A surprise index was 9 P Slovic, D Kahneman, and A Tvsersky.

Judgment under uncertainty: Heuristics andbiases. Cambridge University Press, NewYork, 1982

then calculated based on the percentage of actual values exceedingthe estimated bounds. If the experts had properly estimated thehigh and low bounds, the index should be 2%. But for most thesurprise index was 40% or greater, i.e. they were overconfidentin accuracy, resulting, as Kevin Driscoll points out in the quotebelow, the breakdown of intuitive or experience based estimationsof probabilities for rare events.

“20 years of engineering experience, is usually less than 5,000 hoursof real hands-on system experience and usually negligible field expe-rience...So, when a designer says that the failure can‘t happen, thismeans that it hasn‘t been seen in less than 5,000 hours of observa-tion...5,000 hrs is a few days of flying time for a popular aircraft typeand one days drive time for a popular car type...We cannot rely onour experience-based intuition to determine whether a failure canhappen within required probability limits”

Kevin Driscoll Murphy was an Optimist

If we consider Driscoll’s example and compare the experiencegained over 5,000 hours with the safety requirements for catas-trophic accident rates of 10−9 per hour we can estimate the cumula-tive probability of an expert (engineer) seeing such a failure in theirentire career is approximately 10−6 or one in a million. This leadsin turn to a form of normalcy bias amongst experts, such as theengineers of Driscoll’s example, and their rejection of the possibilityof failures as ‘not credible’.10 Compounding the problem of a lack 10 Normalcy bias is an extreme form of the

availability heuristic where the absenceof prior example causes a failure to evenacknowledge the possibility of an eventoccurring because it has never happened,and therefore to prepare for it.

of familiarity is that when such events do occur they are not actuallyrecognised as being significantly different and worthy of study. Forexample the fact that there was a systemic software problem with


the Therac 25B medical radiation machine was not identified fora number of years simply because there was no organisation tocollate such data from the individual hospitals in which incidentsoccurred and each hospital treated their over-doses as isolatedincidents.

1.4.3 A short discourse on Bayes

“...the theory of inverse probability (Bayesian probability) is foundedupon an error, and must be wholly rejected.”

R.A. Fisher Statistical Methods for Research Workers

The apex of subjectivism is Bayesian reasoning, which is simplyabout how we can and should revise our beliefs in the light of newevidence. Bayes Theorem itself is a simple mathematical expressionrelating the probability of A given that B is true, e.g P(A | B),to the probability of B given that A is true, e.g P(B | A), giventhe individual (independent) probabilities of A and B and thatP(B) 6= 0.

P(A | B) =P(B | A) P(A)

P(B)(1.2)

If we’re trying to prove a hypothesis (H) based on evidence (E) wecan substitute H and E for A and B respectively to give.

P(H | E) =P(E | H) P(H)

P(E)(1.3)

For Bayesianism beliefs must 1) originate with a numerical proba-bility estimate, 2) adhere to the rules of probability calculation, and3) follow an exact rule for updating belief estimates based on newevidence. The term P(E | H) represents the likelihood function thatis the probability of the observed data arising from the hypothesis.The term P(H) represents our prior, reflecting our state of beliefbefore the data turns up, finally the term P(H | E) is our poste-rior term that represents the probability of the hypothesis giventhe data.11 Because the bayesian approach integrates prior belief 11 The use of priors and their subjectivist in-

terpretation is where frequentists (violently)disagree with subjectivist interpretations.

(via the prior) the two approaches, frequentist and bayesian, cangenerate significantly different answers to important questions.

A comparison illustrates the difference between bayesian and fre-quentist approaches. Say we have a gifted musician as an acquain-tance who tells us that they can reliably pick the composer of a workbased on one stanza, we test them and they do so eight out of tentimes correctly. We have another acquaintance who tells us thatthey can reliably predict whether a coin will fall head or tails, againwe test them and find they are successful eight out of ten times. Ifwe consider the reliability of either a frequentist would state that,based on the data, it was 0.8 for both cases. However a bayesianwould integrate their prior belief as to the relative predictive powerof both acquaintances and come up with reliability measures thatdifferentiates between the two.

what is risk? 23

Bayesian reasoning also allows us to address biases such as the pros-ecutors fallacy which result from ignoring the base rate frequency.

1.4.4 The general problem of imprecision

“...important responsibility not to use numbers, which convey theimpression of precision, when the understanding of relationships isindeed less secure. Thus, while quantitative risk assessment facili-tates comparison, such comparison may be illusory or misleading ifthe use of precise numbers is unjustified.”

National Research Council, Governing Board Committee on theAssessment of Risk, 1981, p. 15.

There is an implicit requirement on estimates of the probability ofsome event, and that is that we understand the event in enoughdetail that we can actually determine it’s probability. This is fairlyeasy to do with games of chance, but becomes more problematicwhen we start to consider even moderately complex future eventsin the real world12. For example if we were to ask a person to 12 A study of professional bridge play-

ers found them to make near perfectlycalibrated probability assessments.

define the probability of a serious road accident the probabilityof such an event would be affected by that person’s subjectivejudgement of what constitutes serious. If we do not specify theevent precisely enough that a set of observers if given all availableinformation would agree it had occurred then our probabilityestimate will likewise contain uncertainty to the extent of theimprecision of the definition.13 A practical illustration is provided

13 the clairvoyant test heuristic. One way toassess how well specified an event or quan-tity is is to imagine a clairvoyant who couldknow all facts about the universe, past,present and future. Give the description ofthe event could the clairvoyant say whetherit has occurred, or will occur or state theexact quantitative value. If the answer isthat yes, in principle, they could predict theevent the it is well specified.

in Table. 1.1 below of the importance of well specified events in riskassessments. In the example the effects of imprecise definition ofthe Top Level Event (TLE) of a fault tree.

Fault Tree Top Level Event Degree of clarity

Fuel leak Initially vague definition

Fuel leak causes a potentiallyexplosive build up of propel-lant

Event is now bounded as tothe consequences that weactually care about

Fuel leak sufficient to cause apotentially explosive build upof propellant (20 ppm) in theAPU module while the systemis shutdown for a nominal 5

day mission period

Exposure context established,risk threshold value quantifiedand a duration of exposuredefined

Table 1.1: Example of increasing clarity inthe defintion of an event. Note that as weadd independent factors to the definition wealso decrease the combined priority of theTLE, a balance therefore has to be struckbetween representativeness and specificityof the TLE.

Conversely where the degree of uncertainty and ambiguity maymake it impossible to clearly specify the event that we are inter-ested in then clearly attaching a quantitative measure to such avague and ill-specified event imputes a degree of precision andauthority that is lacking.


1.4.5 The problem of linguistic imprecision

The problem of imprecision is further compound by the problem oflinguistic ‘sloppiness’ when we discuss probabilities using wordsrather than numbers. Wallsten et al. performed a series of experi-ments14 that demonstrated there is significant overlap and variance 14 Thomas S Wallsten and et al. Measuring

the vague meanings of probability terms.Journal of Experimental Psychology:General, 115(4):348–365, 1986

in terms of probability ranges that when they are assigned a quan-titative measure15. Figure. 1.4 is derived from Wallsten’s paper

15 Our understanding of probability isinevitably firmly rooted in a specifictemporal context. For example a qualitativestatement that ‘rain is likely’ will have adifferent probability meaning to a residentof Dublin (today) than to a resident of AliceSprings (this year).

illustrates the degree of variability ins such linguistic expressions ofprobability.

0 0.2 0.4 0.6 0.8 1

Almost impossibleDoubtful

ImprobableUnlikely

TossupPossible

Good chance

LikelyProbable

Almost certain

Probability

Figure 1.4: Wallsten et. al studied thevagueness of subjective probability terms.Note the considerable variability, witheven the median value of ‘tossup’ havinga range from 0.4 to 0.6 as well as theenormous difference between subjects forthe term possible. Range bars are betweenthe median high and low values for thatterm with error bars indicating the 25% to75% percentile range for their associatedhigh/low values.

The imprecision of subjective terminology and concepts can alsoaffect communication, for example what exactly is meant by ’eventX is possible’ in terms of probability. Yet another way in which am-biguity can creep in is if the phrase “at this time” is used withoutdefining the duration. Assuming a short time frame may lower aprobability, say assuming an hour or a day, whereas the probabilityis greater if we are assuming some greater period of exposure, suchas a month or a year. We should be careful to minimise the effectsof linguistic imprecision where we can through the use of preciselydetermined terminology and quantitative measures to define theirbounds.

1.4.6 The problem of determining probability

“...an accident can seldom count higher than three ... which is amystery of probability that my intuition tells me is rooted at the verybase of physics.”

Mark Helprin, A Soldier of the Great War

Demonstrating via classical statistics the very low probabilities ofaccidents requires extremely long durations, which as a practicalmatter are unlikely to achieve before we place a system into ser-vice16. Instead we fall back on estimating the probability of more 16 Bev Littlewood and Lorenzo Strigini.

Validation of Ultra-High Dependability forSoftware-based Systems. Springer BerlinHeidelberg, Berlin, Heidelberg, 1995

frequently occurring precursor events, such as component failures

what is risk? 25

or operator errors, and combining them into a logical model thatcan be used to compute the probability of the accident of interest.Of course this relies on us knowing the set of precursor events andtheir probabilities with enough precision to be able to calculate aquantitative value as well as the correctness of the model17.

17 A more subtle problem is that the inherentfocus of building such models tends to beupon the estimation of probabilities for anexisting technological system, else howcould we derive such quantitative datawhich takes the severity of the outcome asa given of the technology.

• The first problem is that identifying such precursor events andstates for novel or complex technologies is a non-trivial exercise.Many causal factors are either ambiguous or debatable.18 and 18 For example it was not obvious before the

Germanwings tragedy that the combinationof more secure cockpits, introduced inresponse to the 9/11 attacks, with twopilot operation would increase the risk of asuccessful pilot ‘suicide by jetliner’

there is unfortunately always the possibility of incompleteness.

• The second problem we face is in estimating the probability ofthese precursor events. Some probabilities may seem relativelystraight forward to estimate, e.g. we may have empirical data forsimple component failures. But others, such as human error orsoftware faults, are much more complex. A series of assumptionsmay be necessary19 in order to make the model ‘work’. 19 Such as assuming that people will

operate the system in accordance withthe rules, that events are independent orthat we can model human error as a fixedrandom failure rate

• Thirdly our model may incorrectly capture the interaction be-tween it’s internal elements or the surrounding environment.

• Finally when we build a model we inevitably simplify by omit-ting details that we don’t believe are salient to the purpose of themodel. A model is a deliberately simplified abstraction of realityand thereby always incorrect to some degree but hopefully not inways that affect the fidelity of the answer we seek. The decisionto include or omit detail is inherently a quesion that requiresjudgement, introducing subjectivity into the modelling process20. 20 In the Zion Nuclear Power Plant (NPP)

PRA model studied by Rasmuson & Veselyit was found that modifications to the modelto explicitly include modelling uncertaintyhad significant effects on low probabilityestimates. For affected events and failureswith probabilities in the vicinity of 10−6

per year the best estimates (means) weresometimes changed by factors of 10 ormore and the error factors were increasedby a factor of 2 or more.

W E Vesely and D M Rasmuson. Uncertain-ties in Nuclear Probabilistic Risk Analyses.Risk Analysis, 4(4):313–322, December1984

Because of the limitations of our knowledge we inevitably fallback upon subjective probability estimation based on an assessorsbackground (subjective) knowledge, of course the assumptions anduncertainties of this knowledge should also be evaluated along withlikelihood keeping in mind the law of comparative crudeness i.e thattotal uncertainty is dominated by the worst individual estimate,usually those relating to human error rates.

1.4.7 Probability and the analytical boundary

Unlike the games of chance that Fermet and Pascal concernedthemselves with the systems we’re interested in are open, and inorder to determine risk we must make some sort of decision as towhat parts of the environment we should include or not in our riskanalysis. Such decisions naturally introduce an irreducible degreeof subjectivity into any risk assessment and the possibility that wehave omitted something from our model that we ought not to. If wehave made that assumption then that absent element may (or maynot) materially effect the subsequent risk assessment. For examplethe effects of plant flooding was not included in the PRA of theFukushima Daichii NPP.21 21 Woody Epstein. A Probabilistic Risk

Assessment Practioner looks at the GreatEast Japan Earthquake and Tsunami.Whitepaper, Ninokata Laboratory, TokyoInstitute of Technology, April 2011


1.4.8 The incompleteness problem

Probabilistic logic inherently requires a defined set of events andtheir associated probabilities. But what about the risk posed byevents which we did not anticipate and for which a likelihood (andseverity) cannot therefore be assigned? How do we know that wehave identified and considered all risks? For many years the USNuclear Regulatory Commission (NRC) required the modelling andquantitative risk assessment of major coolant piping breaches aspart of the safety assessment of new nuclear plants. In the wake ofthe Three Mile Island (TMI) accident it became apparent that smallscale leaks could also result in significant core damage events andto that extent the NRCs prior risk assessments had been incompleteand underestimated the operational risk for the fleet of nuclearplants.22 22 Conversely in the post mortem of TMI it

was also apparent that the reactor core andlast ditch containment systems performedfar better than had been anticipated.

The heart of the problem is that if we perform only one analysiswe don’t know whether we have identified all the risks In factthere’s good empirical evidence to indicate that there are signif-icant incompleteness problems in the risk identification process.In a comparison of Hazards and Operability Study (HAZOPS),Action Error Analysis (AEA) and Work Safety Assessment (WSA)applied to a common analysis of a chemical processing activitySoukas23 found that a number of deviations24 and determining 23 J. Soukas. The limitations of safety

and risk analysis. In Institute of ChemicalEngineers Symposium, 11. Institute ofChemical Engineers, March 200724 HAZOPS uses the term deviation,indicating an event or state that varies fromthe nominal plant state, in concert withdetermining factors, representing stableplant properties, as an operationalism ofrisk.

factors unidentified by the original HAZOPS methodology. In theidentification of deviations HAZOPS achieved 77% coverage whilefor determining factors achieved 75% coverage. The additionalsources of risk contributed 38% percent of the total probabilityof an accidental release during a storage-loading operational sce-nario for a liquified sulphur dioxide product, a not insignificantunidentified risk one might assume.

Factor HAZOPS AEA WSA Total

Deviation 65 16 3 84

Determining factor 12 3 1 16

Table 1.2: Incompleteness of HAZOPS riskanalysis. HAZOPS identified factors, factorsidentified by AEA alone and WSA alone.

Reframing the problem of completeness we might view the sit-uation as one in which there exists a hidden population of risks.Any risk identification is we suspect an incomplete sample of thishidden population, if we sample this population again then thedegree of overlap between the two samples would indicate thedegree to which a set of unidentified hazards still exist. A smalloverlap would indicate there remains a large degree of incomplete-ness, while a smaller overlap would indicate less incompleteness.To do this more formally we should carry out at least two separateattempts to identify risks in the system, and under certain assump-tions, we can apply what is called a capture-recapture analysisto derive an estimate of the total number of risks in the hiddenpopulation.25 25 The term capture-recapture reflects the

origin of the technique in the zoologicalsciences where the problem was how toestimate a total (hidden) population from afew samples. However the techniques hasfound wide use in circumstances where wehave only limited knowledge of a hiddenpopulation.

D G Chapman. Some Properties ofthe Hypergeometric Distribution withApplications to Zoological Censuses.University of California, 1951

Formally, for a two sample scenario if X1 is the number of risks

what is risk? 27

identified with the first analysis, while X2 is the number identifiedby the second and if X12 is the number of risks that both identify,then the estimator for the total number of risks will be using theChapman estimator:

N̂ =(X1 + 1)(X2 + 1)

X12 + 1− 1 (1.4)

The capture efficiency of each analysis is:

Ei =Xi

N̂(1.5)

While the variance of N̂ is estimated by:

var(N̂) =(X1 + 1)(X2 + 1)(X1 − X12)(X2 − X12)

(X12 + 1)2(X12 + 2)(1.6)

Giving a 95% confidence interval (assuming normality of N̂) of:

N̂c95 = ±1.965× (var(N̂))0.5 (1.7)

The smaller the overlap between analyses, represented by X12,the greater the indication of a hidden population. UnfortunatelySuokas did not include X12 values in his paper, however capture-recapture analyses have been performed Potts et al. on the validityof two risk identification methods HFMEA and Structured What IfTechnique (SWIFT). Caseley, Guerra and Froome compared hazardsets generated by two separate HAZOPS studies using capture-recapture analysis and the results are summarised in Table. 1.3.Ishimatsu et al. performed a comparison of the System TheoreticAccident Model and Processes (STAMP) against those obtainedusing FTA and upon which a capture-recapture analysis could beperformed26 26 H.W.W Potts, J.E. Anderson, L. Colligan,

P. Leach, S. Davis, and J. Berman. As-sessing the validity of prospective hazardanalysis methods: a comparison of twotechniques. BMC Health Services Research,14(1):41, 2014. ISSN 1472-6963. DOI :10.1186/1472-6963-14-41. URL http://dx.

doi.org/10.1186/1472-6963-14-41;P R Caseley, S Guerra, and P Froome.Measuring hazard identification. In SystemSafety, 2007 2nd Institution of Engineeringand Technology International Conferenceon, page 6 pp. IET, 2006; and T Ishi-matsu, N Leveson, J Thomas, M Katahira,Y Miyamoto, and H Nakao. Modelling andHazard Analysis Using STPA. In Conferenceof the International Association for theAdvancement of Space Safety, Huntsville,2010

Analyses X1 X2 X12 N1,2 N̂ N̂c95

SWIFT-HFMEA 61 72 23 110 188 48

HAZOP-HAZOP 20 30 12 38 49 12

FTA-STAMP 5 17 5 17 17 -

Table 1.3: Capture-recapture statistics forindependent risk (hazard) identificationmethods using Chapman’s estimator. Notethat Pott’s original X1,2 data was expressedwith confidence intervals.

Referring to Table. 1.3 we can see that in no case did the two meth-ods identify identical sets of risks. In all but the FTA-STAMP caseone or other of the analyses identified additional risks indicatingthe presence of a residual hidden population. We should also notethat one of the assumptions of the Chapman estimator is that themethods have an identical likelihood of finding a risk, only inCasely, Guerra and Froome’s HAZOPS example could we be con-fident that this was the case as in the other two examples we arecomparing different methods.27 27 Where N>2 risk identification samples are

conducted a Schnabel estimator may besubstituted for Chapman’s estimator.

All of which poses a distinct problem for the concept of classicalrisk acceptance because, as Figure. 1.5 illustrates with any set of

http://dx.doi.org/10.1186/1472-6963-14-41

http://dx.doi.org/10.1186/1472-6963-14-41


risks that we perceive and explicitly accept we are also implicitlyaccepting an unknown number of unidentified risks. This under-cuts classical theoreies about making decisions under uncertaintywhich rely upon the characterisation of uncertainty as probability.The application of capture/recapture methods is one approach toderive an understanding of the degree of uncertainty about the truedegree of residual risk.

Total risk

Unidentified

Implicitlyaccepted

Identified

EliminatedMitigatedresidual risks(Acceptable)

Explicitlyaccepted

Figure 1.5: Incompleteness and riskacceptance. When we accept risk we arealso implicitly accepting an unidentified setof risks as part of the residual risks. Someof these unknown risks may also be highrisks, we simply do not know.

Discussions about risk acceptance normally revolve around whethersome degree of risk is ‘acceptable’ as assessed using a particularcriteria. However when we accept a known risk we may also acceptsome degree of unknown risk along with it. Unknown risks maydisclose themselves in the life of a system or technology, but somemay never be identified. There is a subtle but significant differencebetween the statement ‘all risks have been mitigated’ and ‘allidentified risks have been mitigated’.

1.4.9 The uncertainty severity relation

The calculation of risk relies on us knowing the set of loss eventsand their probabilities with precision a sufficient to be able tothen calculate a specific value. This is rarely the case for novelsituations or technologies, where many factors are ambiguous oropen to interpretation. As Table. 1.4 illustrates the evidence to dateis that our certainty decreases with increasing severity, which is notencouraging.

Even where we do have statistical data we face the problem thatrisk is expressed as some proportional rate, i.e. N accidents per unit

what is risk? 29

Event Type Occurrence rate Error(E/I) (per year/demand) factor

Common failures I > 1 1.3-2Higher value failures I 10−3

10

Single human errors I 10−310

System unavailabilities E 10−4 - 10−54-10

Accidents E 10−3 - 10−53-6

External events I 10−3 - 10−610-30

Unlikely pipe ruptures I 20-30

Multiple human errors I 20-30

Rare (extreme) accidents E 10−720-30

Table 1.4: Declared uncertainties of PRAprobability estimates expressed as errorfactors, where the error factor is the ratioof the upper 95th percentile or confidencevalue to the median or 50th confidencevalue. Events are either Initiating (I) or Endstates (E). Summary results derived froma study of four US (Arkansas,Big Rock Pt,Zion and Limerick Nuclear Power PlantPRAs by Vesely & Rasmuson.

of exposure to loss.28 The problem is that not all such proportions 28 Exposure could be measured in time, orusage cyclesare created equal, a small ensemble of systems will operate for

fewer hours compared to a larger ensemble and because of naturalvariance you’ll therefore most likely see them occupying both thehighest and lowest loss rate positions in any such comparison ofsafety because, as we divide that number by the total number ofsamples to get our proportional rate, the proportional discrepancyor variance is going to increase as our sample gets smaller. This isknown formally as the standard error on the mean, otherwise knownas de’Moivre’s equation.29 What de’Moivre found was that the size 29 de’Moivre’s equation in this form applies

to binomial processesof a typical discrepancy (variance or σ) in the mean of a sample isproportional to the square root of the number of samples (n), i.e

σx̄ = σ/√

n (1.8)

Formally Equation. 1.8 tells us is that the variability (variance)in the averages (means) of samples taken from a population getsgreater as our sample size n decreases. Conversely as a sample sizeincreases the variance in the mean will decrease and de’Moivre’sequation has significant implications when we try to infer the safetyof systems from empirical data.30 We would therefore expect that 30 An analogy is if we throw a stone into

a small pool of water then the splash willbe comparitively large, but if the pool isvery big then the same stone will makecomparitively less of a splash.

any large set of data about what’s already happened will tend todilute the effect of new data and the effect of one event, such as onedisastrous accident, will become proportionally smaller. Convereslythe smaller such sets of data are the greater the effect of new datacan be.

Figure 1.6: Air France flight 4590 on fireand moments before it’s fatal crash.Image source: Toshihiko Sato

case study: concorde’s safety record. For most of it’s careerthe Concorde aircraft was the the safest aircraft flown because it hadnever suffered a catastrophic accident but then after a single crashit suddenly became the least safe passenger aircraft. How couldConcorde go from being the safest aircraft to the least safest withinone day? By retirement Concorde had logged only 78,073 departuresgiving a point estimate of fleet accident rate of approximately 1.2×10−5 per departure, 14.6 times that of the 747-400 fleet’s accident rateas a point of comparison.31 31 Boeing Aviation Safety. Statistical sum-

mary of commercial jet airplane accidents(1956-2015). Statistical report, BoeingAircraft Company, Seattle, Washington,2015

To understand what happened to Concorde’s exemplary safety


record we turn to de’Moivre’s equation. In the case of Concordethere simply wasn’t a big enough set of data and thus one eventsignificantly effected the proportional rate. So a single Concordeaccident could have a disproportionate affect on the aircraft’ssafety statistics when compared to Boeing’s 747 fleet with it’s muchlarger data set. Sample frequency only tends towards an accuratepredictor of probability as the size of the sample set becomeslarge and this tendency, called the law of large numbers, meansthat small sample sizes, like that of the Concorde can be verymisleading.32 32 Nassim Taleb points out that for heavy

tailed distributions which may apply tothe systems we are interested in samplemeans are unstable, i.e. one single extremeobservation can make it jump, while alsounderestimating true means.

Pasquale Cirillo and Nassim Nicholas Taleb.Expected Shortfall Estimation for ApparentlyInfinite-Mean Models of Operational Risk.SSRN Electronic Journal, 2015

1.4.10 The assumption of independence

Often when we calculate the probability of an event from causalfactors we are dealing with a number of separate factors that allhave to occur. We therefore multiply their probabilities togetherto obtain the total probability as Equation. 1.9 illustrates below.The mathematical and practical implications of this are obvious, aswe increase the number of causal factors we decrease the overallprobability given that Pi ≤ 1 must always be true.

Pt = P1 × P2 × ...× PN =N

∏i=1

Pi (1.9)

Underlying Equation. 1.9 is the assumption that these individualcausal factors are in fact independent. Yet even making what ap-pear to simple assumptions about the independence of events canhave lead to significant errors as the following example illustrates.

case study: the sally clark trial. In one infamous Britishcriminal trial a medical expert testified that because the likelihoodof a child dying of cot death was 1 in 8,543, then the likelihood oftwo children from the same family dying would be the multiple i.e.1 in 73 million and the accused, Sally Clark, who was also a solicitor,was convicted on that basis. The case was eventually overturnedon appeal but not before Sally Clark spent three years wrongfullyimprisoned.

So was the expert in error? Yes, as it turns out the expert’s pas-calian probability was flawed because he overlooked that the twoevents (the two cot deaths) are not random and independent events.Children from a single family living in the same house share arange of common genetic and environmental factors so that if aprior death has occurred then another death becomes statisticallymuch more likely.

The assumption of independence also has significant implicationsfor the safety of technological systems. Designers often opt for theuse of redundant (N > 1) components to provide some neededservice. This is based on the assumption of independence, which ifit is true means that the total probability of failure Pt is provide an

what is risk? 31

exponential decrease in the systems’s failure rate, i.e. for Pi = P thesystem failure rate becomes Pt = PN for N degrees of redundancy.But that assumption is only valid if the redundancy creates nocommon mode failures and no latent (undetected for unknowntime intervals) failures of redundant paths that aren’t currentlyoperating.33 33 Any design that utilises redundancy

must either discharge a claim that suchredundancy is truly independent or lesserclaim to that such dependence can becharacterised and quantified.

With de’Moivre’s formulation of risk there also goes an implicitassumption that P and S can be treated as independent variables, Ris the dependent, but this may not always the case, for example:

• The problem of defining risk as an ensemble statistic is that weactually live in a time series world, were what happens in thepast can and does have a significant effect upon the future:

– a severe accident may curtail a system’s life, truncating theperiod over which we determine our accident frequency andincreasing the subsequently calculated probabilistic risk, as inthe case of the Concorde.

– After an accident we may act to reduce the risk, thus retro-spectively invalidating our initial assessment that the risk wasacceptable.

• both probability and severity may be affected by a single factor.For example excessive speed when driving can increase both theprobability of a crash as well as it’s subsequent severity.

• Our uncertainty over both the likelihood and severity of eventsis related to the severity of such events. That is as the severity ofsuch events increases, our uncertainty as to how probable andhow severe they could be increases.

The assumption therefore that P and S are neatly independent is auseful one, but perhaps represents risk as simpler than it really is.

1.4.11 The assumption of consistent condition

Even if we do have empirical frequencies, the question remainsas to whether we can apply it to predict the probability of futureevents. To do so we must assume, because we cannot prove, thatthe future will be consistent with the past.34 Nor can we necessarily 34 The assumption of consistent condition

is often unstated in probabilistic riskanalyses.

assume that all of the conditions which contributed to the originalresults were recorded or even noticed.35

35 Richard P Feynman and P Feynman.The Meaning of It All. Thoughts of aCitizen-Scientist. Basic Books, April 20091.5 The problem of severity truncation

“The risk equals expectation fallacy. Predicted probability distri-butions for extreme events may deviate strongly from observedoutcome distributions due to the limited value of historical data.Average (expected) outcomes may also be dominated by a very fewextreme outcomes”

Terje Aven Misconceptions of Risk


If we accept accept the risk as expectation of loss we are faced withthe question as to whether when we have summed all individualsources together we are dealing with a finite risk value. Clearly ifwe cannot bound a risk value it’s unlikely that the resultant infinite,or possibly infinite, risk would be acceptable to a rational decisionmaker. Given that extreme losses will dominate the ensemble ofrisks if the have even small finite values in order to use risk ina practical sense we must be able to bound the severity in someway36. 36 Severity truncation also applies as we

reduce the severity, if we include a multitudeof extremely small risks we again increasethe overall risk.1.5.1 Risk and severity truncation

If the probability distribution of our events is normal, or there’ssome other physical limit on severity37 severity truncation may 37 For example, the maximum number of

passengers on a train would notionallyplace a physical constraint on how severeany subsequent train disaster might be.

be relatively easy to argue. But if the distribution is heavy tailed,i.e. there’s a significant probability of extreme events in the tail,it’s very hard to bound the risk because of uncertainty about whatactually is the worst case38. As these sort of extreme events in the 38 What is not often realised is that in heavy

tailed variables, the mean is almost entirelydetermined by extremes. If you are thenuncertain about the tails you are alsouncertain about the mean.

tail also dominate the risk truncating the distribution arbitrarilyby assuming a worst credible case translates into an error in theoverall risk, and as the severity of such events dominates the overallensemble we will be underestimating the risk significantly. Risk, orthe expectation of loss, it should be remembered is a probabilityweighted average, but an average only make sense if there is acentral tendency in the population.

As part of a paper on the construction of risk matrices Swallomcollated the US Army H-60 Blackhawk accident data of Fig. 1.7into a risk curve for the ensemble of classes of accident severity, theresults illustrate that Class A accidents (having a severity of >$1M)outweigh all the other collected accidents in the ensemble of risks39. 39 D Swallom. Mathematical Techniques

to Improve the Utility of a Hazard RiskMatrix. In th International System SafetyConference, pages 1–10. ISSS, 2011

The possibility for so called Dragon king events40, extreme outliers

40 Didier Sornette. Dragon-Kings, BlackSwans and the Prediction of Crises. SSRNElectronic Journal, 2009

whose severity lies well above even that predicted by a heavy tail,present an even greater challenge as even one such outlier event caninvalidate the ensemble average.

1.5.2 Worst case or worst credible?

The problem of establishing a finite risk value becomes intractablewhen we believe we are dealing with distributions whose tails areheavy enough to prevent them from having a finite mean, and inturn that the the risk is infinite as the shape parameter ξ > 1.41. If 41 See for example the Pareto distribution

of reactor accident severity which results ina theoretically unbounded, and thereforeuninsurable, risk

Marius Hofert and Mario V Wuthrich.Statistical Review of Nuclear PowerAccidents. SSRN Electronic Journal, 2011

we accept the infiniteness of the mean then a single loss can alwaysbe so large as to cause ruin, termed the dismal theorem by MartinWeitzman42. In reality all losses are bounded even if the severity

42 Weitzman believes that in these cir-cumstances standard actuarial analysesbecome useless, he gives climate changeas a canonical example.

is so large as to make it unobservable to us, and therefore our lossdistribution will be truncated at some point. Realisation of this is re-flected in the prevalence of the use of terms such as ‘worst credible’and ‘worst possible’ in risk assessments as an analytical fiddle thatallows the analyst to bound the severity distribution for heavy tail

what is risk? 33

103 104 105 106 107

10−7

10−6

10−5

10−4

10−3

1

10

100AB

C

D

Accident cost ($S)

Prob

abili

tyof

acci

dent

(P)

per

fligh

tho

urUS Army aviation accident ensemble Figure 1.7: US Army aircraft Class A,B,C

and D accident risk ensemble plottedonto iso-risk contours. As total risk RT =RA + RB + RC + RD we can see that theClass A mishaps dominate the ensemble.

distributions. Even when the problem of truncation is specificallyaddressed in the analysis difficulty remains in determining where toplace such a truncation and how to do it in a defensible fashion.43 43 Of course the practical remedy to such

analytical heartache is to actually bound theseverity in the real world.

More credibly one can calculate what Cirillo and Taleb term theshadow moments44 of the distribution thereby rescuing the concept

44 Pasquale Cirillo and Nassim NicholasTaleb. Expected Shortfall Estimationfor Apparently Infinite-Mean Models ofOperational Risk. SSRN Electronic Journal,2015

of risk for heavy tailed distributions and allowing us to calculate afinite if extreme value.

case study: air accident severities. Fig. 1.8 illustrates the prob-lem we face in determining the truncation value even when we havestatistical data. The graph represents the cumulative frequency ofthe severity of US classified FAR Part 21 classified aircraft accidentsfor the period 1966 to 2013, we can see from the graph what lookslike a truncated power law with an exponential cut-off. If we were totreat the most extreme value45 of the plot as an outlier and exclude it 45 The Tenerife air disaster where two B747

aircraft collided with the loss of 574 lives.we would probably set our cut off value H ≈ 300. However shouldwe truncate the tail in such an arbitrary manner? Should we con-sider it as part of the distribution? Or should we instead treat it as adragon king i.e. as belonging to a different statistical population withuniquely extreme properties46. Unfortunately with only a sparse 46 Didier Sornette. Dragon-Kings, Black

Swans and the Prediction of Crises. SSRNElectronic Journal, 2009

data set and no prior knowledge of the distribution, answers to suchquestions very quickly become questions of judgement.

When considering what the truncation threshold should be set atwe also need to consider again that future may not be the same asthe past. In the case of aviation the carrying capacity of airframeshas increased which in turn increases the potential severity of acci-dents47. Had we established a truncation value for the distribution 47 The 1962-1972 period saw the capacities

in mixed class layouts reach 200+ with the727 then climb past 400+ with the 747 inthe early seventies.

in the early 1970’s we would have found it exceeded in subse-quent decades. The problem of arbitrary tail truncation leads tothe possibility that the probability of extreme events is significantlyunderestimated or discounted completely.48 In the case of aircraft 48 This is sometimes called the distribution

game.accident severity we might discount the probability that an aircraftcollision involves more than two aircraft, until we consider the


100 101 102 103

10−2

10−1

100

H

Tenerife

Fatalities per accident (S)

Cum

ulat

ive

freq

uenc

y(c

.f)accident severity vs cumulative frequency

1962-2013

1962-1972

Figure 1.8: Log-log plot cumulative fre-quency of US FAA Part 21 classified aircraftaccident severity 1962-2013. A cumulativefrequency histogram or rank/frequencyplot is used rather than a simple frequencyhistogram as it better preserves informationin the tail of the distribution where the num-ber of samples in the bins becomes smalland statistical fluctuations are thereforelarge as a fraction of sample number. Therank/frequency plot also displays power lawrelations as a straight line on a log/log plot.

events of the 7th July 2017 in which a Canadian Air A320 narrowlymissed landing on four other passenger aircraft on a taxiway.

case study: fukushima and severity truncation. In re-sponse to the Fukushima Daichi nuclear crisis the UK governmentestablished an expert committee chaired by the Government’s chiefscientist to provide advice to government on the ongoing crisis. Thecommittee initially calculated what was believed at the time to bea credible worst case scenario for the disaster. However after it be-came apparent that a considerable inventory of spent fuel rods couldalso become exposed the committee revised their initial worst caseprojections to what they called the enhanced worst case.49 49 Due to damage to the fuel rod cooling

ponds.

Figure 1.9: Fukushima daichi reactordamage.

Image source: Air Photo Service/AP.

1.5.3 Arguing implausibility

In 1943, the French mathematician Émile Borel published a booktitled Les probabilités et la vie, in which he stated what has come tobe called Borel’s law which might be paraphrased as, “Events witha sufficiently small probability never occur.”50, to quote Borel for a

50 Borel’s law (actually a heuristic) has alsobeen called the infinite monkey theorem inwhich the time it would take a room full ofmonkey’s on typewriters to write the worksof Shakespeare is used as a metaphor forimplausibility.

moment:

“Such is the sort of event which, though its impossibility may notbe rationally demonstrable, is, however, so unlikely that no sensibleperson will hesitate to declare it actually impossible. If someoneaffirms having observed such an event we would be sure that he isdeceiving us or has himself been the victim of fraud.”

Émile Borel Les probabilités et la vie 1943

Borel’s law may be thought of as another avenue by which wecan advance an argument that catastrophic hazards are not justunlikely but simply will not occur. In effect when a safety studyis published with a probability of accident incredibly low theauthors are in effect arguing that, as Borel observed, that the event

what is risk? 35

is so improbable that it is functionally equivalent with impossibleand we may reasonably conclude it will never occur. Of coursewhere we argue, as is the case in the safety assessment of the EPAreactor quoted by Ramana51, that the improbability is based on the 51 M V Ramana. Beyond our imagination:

Fukushima and the problem of assessingrisk. April 2011

combined probability of numerous independent events then weneed to have great trust in both their actual independence as well asour calculations.

1.6 The irreversibility paradox

“Never cross a river if it is on average only 4 feet deep.”

Nasim Taleb Debate with P.Jorion 1997

Returning to the fundamental concepts of classical probability for amoment the way in which Pascale and de’Moivre dealt with uncer-tainty over future events was to imagine a set of parallel worlds. Toassess the risk/expectation of some uncertain venture, an averageis taken across those parallel worlds. Thus the classical conceptionof risk as an expectation is fundamentally a population or ensemblestatistic, which when used in the context of decisions as to howto bet on a throw of dice, whether to take up an annuity or offerinsurance works quite well. However there’s another context to de-cisions on risk and this is to consider a risk as it affects the decisionmaker in this world only across time. In other words there can alsobe a time series context to decisions about risk. As Taleb points outthe fact that a river’s ensemble average depth is only four feet isnot much use if your concern is stepping into an eight foot deepchannel halfway across. Considering risks from each perspective,time or ensemble, can it seems lead to very different decisions.

A game called ‘Russian dice’ invented by the mathematician OlePeters illustrates this difference. In this game we roll the dice and ifyou get a six, ‘Bang!’, I get to shoot you. Now if you considered theexpected value it would be 3.5 e.g (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5.If we asked 1000 people to roll the dice we’d see that expectedvalue (ensemble statistic) and this would indicate that the riskwas acceptable, but if I asked you to consecutively roll the dice it’svery likely you’d see a six come up and ‘bang’ you never get tobenefit from that ensemble average. Ensemble statistics may workfor populations, but for individuals facing a series of bets they candeliver alarmingly counter-intuitive results.

1.6.1 The St Petersburg paradox

Let’s consider for a moment a special lottery, buying a ticket we getthe opportunity to play a game in which a coin is tossed sequen-tially. Every time the coin comes up heads we win but if it comesup tails the game ends and we collect what’s in the pot. Our initialstake starts at two dollars and doubles with every heads we throw.


E =12× 2 +

14× 4 +

18× 8 + ... = 1 + 1 + 1... = ∞ (1.10)

Thus the expected value of the lottery is infinite and as rationalactors we should, in theory, be willing to pay any amount to buya ticket. The problem is that when people are presented with thislottery they are unwilling to spend more than a small amount onthe lottery ticket. This difference between expectation as a measureof the value of the lottery, i.e. infinite, and the small value weactually place on it is termed the St Petersburg paradox. DanielBernoulli the 18th century mathematician was the first to formulatean answer to this problem.52 Bernoulli argued intuitively that the 52 The paradox takes it’s name from Daniel

Bernoulli’s sometime city of residence,although it was his brother who originallyarticulated the paradox

increase in the usefulness or what he called the utility (U) of yourtotal wealth (w) from a small gain is inversely proportional to thewealth you already have, as our wealth increases the value (orutility) of a gain would therefore decrease. So instead of calculatingthe expected value of the game we should calculate the expectedmarginal utility ∆E(U) of playing. Bernoulli used a logarithmicutility function to express this finite expected marginal utility. Asthe cost of the ticket (c) is now included into the equation we candetermine what value of c, given your total wealth will deliver apositive change in the marginal value making it attractive to buy aticket 53. 53 Daniel Bernouuli. Exposition of a

new theory on the measurement of risk.Econometrica, 22(1):22–36, 1954

∆E(U) =∞

∑k=1

12k [

utility after︷︸︸︷ln(w0 + 2k − c)−

utility before︷︸︸︷ln(w0) ]... < ∞ (1.11)

Daniel Bernoulli’s solution reconciles the mathematics with ourintuitive feeling for how much this lottery is worth to us. But, theproblem is that this is a intuitive solution therefore you can adjustthe expected value of the game to give an even greater pay-off andthe paradox re-appears. The other problem with Bernoulli’s solu-tion is that it while describes a value judgement, it does nothing toexplain the underlying reasons for such a judgement54. Yet despite 54 For completeness of discussion there

have been a number of alternativesproposed to resolve the paradox rangingfrom alternative formulations through tooutright rejection of it’s premises

these loose threads utility theory has remained the standard wayin which to evaluate investment decisions made under uncertainty.The mathematician Ole Peters proposed an alternative solutionto the paradox that does not rely on an arbitrary utility function,instead Peters proposes that the difference results from people’sintuitive understanding that the bet is based on a time series ratherthan a population ensemble and that therefore the expected value(ensemble average) is an inappropriate guide.55 Technically speak- 55 Ole Peters. The time resolution of the

st petersburg paradox. PhilosophicalTransactions of the Royal Society ofLondon A: Mathematical, Physical andEngineering Sciences, 369(1956):4913–4931, 2011. ISSN 1364-503X. DOI :10.1098/rsta.2011.0065. URL http:

//rsta.royalsocietypublishing.org/

content/369/1956/4913

ing the system is non-ergodic and therefore we cannot assume thatthe time series average is equivalent to the ensemble average.

http://rsta.royalsocietypublishing.org/content/369/1956/4913



what is risk? 37

1.6.2 Ergodic and non-ergodic systems

In many ways we can consider our use of a new technology as play-ing a variant of the St Petersburg lottery. If we use our technologysafely we accrue the benefits but should we have an accident (losingour lottery) there can be severe penalties. The question is whetherclassical expectation based risk is, as de’Moivre put it, the ‘truemeasure’ of such risks? Bernoulli’s paradox indicates that perhapsit is not. Say we have a process plant which we know will returnus $100 million dollars per year (R), we also believe that there’s achance of a catastrophic and irrecoverable accident which wouldcost us $100 billion dollars (L) with a probability (Pl) of 1 chance in10,000 per operating year. Should we operate the plant for a year?First let’s consider the answer from the point of view of expectation.Referring to Figure. 1.10 we see that there are two possible alterna-tive futures. In one, operation results in a loss, while in the otherthe plant survives. Calculating the ensemble average using Eqn 1.12

we obtain a positive ensemble average.

Ee = ((1− Pl)× R) + (Pl × L) (1.12)

= ((1− 1104 )× $108) + (

1104 ×−$1011)

= $9.999× 107 − $107

= $8.999× 107

According to the ensemble average we should proceed with build-ing the plant and operating it for one year. This looks good becausewe can extrapolate that as we operate the plant over further yearsour expectation will increase. But this is based on the ensemble av-erage derived from our parallel worlds theory which assumes thatall events in the ensemble are independent, that is we are neglectingthe possible effects of time.

Start

Loss

L Pf

Survival

R (1− Pl)

Figure 1.10: Event tree representation ofoperating of the the plant for one year

Let’s now consider the time average which does not consider thetwo outcomes as independent. This change in averages is necessarybecause the system that we are trying to analyse is not ergodic56

that is it’s not stable about some equilibrium point but in factcan change, sometimes radically, over time.57 Because of this the 57 O. Peters and M. Gell-Mann. Evaluating

gambles using dynamics. Chaos: AnInterdisciplinary Journal of NonlinearScience, 26(2):023103, Feb 2016. ISSN1089-7682. DOI : 10.1063/1.4940236.URL http://dx.doi.org/10.1063/1.

4940236

ensemble average isn’t a reliable predictor of the time average andwe need to actually calculate the time average to understand the

http://dx.doi.org/10.1063/1.4940236

http://dx.doi.org/10.1063/1.4940236


risk associated with operating the plant. In order to calculate the 57 A dynamic system is considered ergodicfrom a probability perspective if when weaverage it’s behaviour over time it turnsout the same as when we averaged overthe space of all the system’s states in itsphase space (i.e. Fermet’s possible parallelworlds).

time average we assume the two possible outcomes occur one afterthe other over two consecutive periods, we multiply these valuestogether and then take the square root to obtain the geometric ortime average. As we have to deal with a loss which is a negativevalue we utilise the following formula for the geometric mean of aentities where m = 1 is the number of negative values and n = 2,the total number.

Et =

(n

∏i=1

ai

)= (−1)

mn exp

[1n

n

∑i=1

ln |ai|]

(1.13)

This gives us, in base 10 for convenience, the following.

Et = −1 exp10

[12(| log10((1− Pl)× R)|+ | log10(Pl × L)|)

]≈ −1 exp10

[12(8 + 7)

]≈ −$107.5

Our time average as it turns out is determinedly negative, confirm-ing both the non-ergodic nature of our system and that from a timeperspective this appears not to be such a good bet after all. If weare dealing with extreme consequences, the time average clearlyprovides a more truthful guide to such risk than the ensemble av-erage. But how do we do put this insight to practical use? Whatwe are interested in is not just what we might gain but also what isan acceptable severity of loss and for this we need to establish ourboundary for how bad it should get, that is is looking at risk fromthe consequential, rather than probabilistic perspective. For this weneed to turn to the work of a contemporary of Claude Shannon atBell Laboratories.

1.6.3 Kelly’s criterion, or not betting the farm

In 1956 John Larry Kelly, then a researcher at Bell Laboratory,published a paper looking at how a gambler facing a series of riskybets could optimise the growth of his wealth in the long run whileavoiding ruin along the way58. Kelly’s answer was that at each turn 58 J. L. Kelly. A new interpretation of informa-

tion rate. The Bell System Technical Journal,35(4):917–926, July 1956. ISSN 0005-8580.DOI : 10.1002/j.1538-7305.1956.tb03809.x

they should only wager a specific fraction of their current wealth, thefraction determined by the odds and potential winnings. From ourperspective we are interested in what constitutes an acceptable orunacceptable risk and the Kelly criteria allows us to decide whetherto commit to a risky course of action based on what constitutesthe threshold of acceptable loss. The Kelly criteria in establishesthe normative boundary between what we might typify as rational(albeit aggressive) and irrational investment decisions. If a decision

what is risk? 39

maker is uncomfortable with the volatility, that is the chance oflarge losses, then a threshold below that of the Kelly’s may beapplied, and there may well be good reasons for doing so.59 59 For example uncertainty over the

parameter’s used to quantify the risk.

f ∗ =bp− q

b=

p(b + 1)− 1b

=(1− q)(b + 1)− 1

b(1.14)

In Eq. 1.14, f ∗ is the fraction of the current bankroll to wager (howmuch you want to chance on a bet), b is the net odds60, p is the 60 Odds are expressed as your potential

winning relative to the amount you wager,e.g. “b to 1” odds means that you would win$b, on top of your $1 wagered for a $1 bet.

probability of winning while q is the probability of losing, naturallyq = (1− p). For our plant we can calculate the net odds as the ratiobetween potential revenue and potential loss b = R/L = 10−3 whileq = Pl = 10−4 is the probability of losing. Giving us:

f ∗ =(1− q)(b + 1)− 1

b

=(1− 10−4)(10−3 + 1)− 1

10−3

= 103 ×[(1− 10−4)(10−3 + 1)− 1

]= 0.8999

As we know what our possible worst case loss is we can use f ∗ tocalculate the our starting wealth (or bankroll) N by multipling ourwager by the reciprocal of f ∗.

N =1f ∗× b

= $1.1112× 1011

To be able to tolerate such losses over the long haul you wouldneed an ability to start with this amount to make it a rationallyacceptable wager in accordance with the Kelly criteria, you canreduce the bankroll (N) by reducing the severity of the loss, orby reducing the probability of that loss. An individual can alsoreduce their N by laying off the bet or insuring the losses, whichtransforms the question into one of whether the insurance pool isdeep enough to handle such a loss.61 61 A rule of thumb is that when one sees

an insurance market developing you aredealing with risks that are considerednon-ergodic by individual decision makersparticipating in that market

1.7 Summary: Classical risk can be an unreliable guide

At this point it might be worth stopping and summarising whatclassical risk is not:

• Despite it’s mathematical trappings, it is not an objective at-tribute, it’s a subjective assessment of our degree of belief, oruncertainty, as to the outcome of some proposition.

• Nor is it directly measurable, instead we define it by the veryoperations we use to identify and characterise it.


• Finally we cannot say if it our understanding is complete, wemay be unaware of our exposure and even if we are awarethere may be unidentified uncertainties in the parameter’s wecharacterise it with, that we are likewise unaware of.

Nor when it comes to extreme events, is classical risk a reliableguide. For such strongly non-ergodic risks we need to turn toalternative consequence focused methods such as Kelly’s. Theclassical definition of risk is not a definition of ‘risk’ itself but ratheran operationalisation of our perception of risk, that applies onlywhen we can guarantee ergodicity. In the following chapter we willexplore further the relation between our perception of risk and ourassessment of it.

Bibliography

Federal Aviation Agency. FAA Advisory Circular: System Designand Analysis. Advisory Circular AC 25.1309-1, Federal AviationAgency, 1982.

Allan Benjamin, Dezfuli Homayoon, Christopher Everett, CurtisSmith, Michael Stamatelatos, and Robert Youngblood. NASA SystemSafety Handbook, v1.0 edition, February 2012.

Daniel Bernouuli. Exposition of a new theory on the measurementof risk. Econometrica, 22(1):22–36, 1954.

Gareth Blalock, Vrinda Kadiyali, and Daniel H Simon. DrivingFatalities after 9/11: A Hidden Cost of Terrorism. Applied Economics,41(14):1717-–1729.

L Bostwick. Development of LOX/RP-1 engines for Saturn/Apollolaunch vehicles. In 4th Propulsion Joint Specialist Conference, Reston,Virigina, February 2013. American Institute of Aeronautics andAstronautics.

Courtney Brooks, James M. Grimwood, and Loyd S. Swenson Jr.Chariots for Apollo: A History of Manned Lunar Spacecraft. NASAHistory Series. NASA, 1976.

Frederick P Brooks Jr. The Design of Design. Essays from a ComputerScientist. Pearson Education, March 2010.

P R Caseley, S Guerra, and P Froome. Measuring hazard identi-fication. In System Safety, 2007 2nd Institution of Engineering andTechnology International Conference on, page 6 pp. IET, 2006.

D G Chapman. Some Properties of the Hypergeometric Distri-bution with Applications to Zoological Censuses. University ofCalifornia, 1951.

Pasquale Cirillo and Nassim Nicholas Taleb. Expected ShortfallEstimation for Apparently Infinite-Mean Models of OperationalRisk. SSRN Electronic Journal, 2015.

Prof Dr Edsger W Dijkstra. On the Role of Scientific Thought. InSelected Writings on Computing: A personal Perspective, pages 60–66.Springer New York, New York, NY, 1982.


Mark Dowie. Pinto Madness. Mother Jones, (September/October),1977.

John Downer. Anatomy of a disaster Why Some Accidents areUnavoidable. Technical Report 61, London School of Economicsand Political Science, London, March 2010.

John Downer. Disowning Fukushima: Managing the credibility ofnuclear reliability assessment in the wake of disaster. Regulation &Governance, 8(3):n/a–n/a, July 2013.

Kevin Driscoll, Brendan Hall, Hakan Sivencrona, and Phil Zum-steg. Byzantine Fault Tolerance, from Theory to Reality. InComputer Safety, Reliability, and Security, pages 235–248. Springer,Berlin, Heidelberg, Berlin, Heidelberg, September 2003.

Kevin R Driscoll. Murphy Was an Optimist. In Computer Safety,Reliability, and Security, pages 481–482. Springer Berlin Heidelberg,Berlin, Heidelberg, September 2010.

Daniel Ellsberg. Risk, Ambiguity, and the Savage Axioms. TheQuarterly Journal of Economics, 75(4):643, November 1961.

Woody Epstein. A Probabilistic Risk Assessment Practioner looksat the Great East Japan Earthquake and Tsunami. Whitepaper,Ninokata Laboratory, Tokyo Institute of Technology, April 2011.

Richard P Feynman and P Feynman. The Meaning of It All.Thoughts of a Citizen-Scientist. Basic Books, April 2009.

Baruch Fischhoff. Cost benefit analysis and the art of motorcyclemaintenance. Policy Sciences, 8(2):177–202, 1977.

Baruch Fischhoff, Paul Slovic, and Sarah Lichtenstein. Faulttrees: Sensitivity of estimated failure probabilities to problemrepresentation. Journal of Experimental Psychology: Human Perceptionand Performance, 4(2):330–344, May 1978.

A P Fiske and P E Tetlock. Taboo Trade-offs: Reactions to Transac-tions That Transgress the Spheres of Justice. Political Psychology, 18

(2):255–297, 1997.

Susie Go. A Historical Survey with Success and Maturity Estimatesof Launch Systems with RL10 Upper Stage Engines. In 2008 AnnualReliability and Maintainability Symposium, pages 491–495. NASAAmes Research Center, IEEE, January 2008.

I J Good. On the Principle of Total Evidence. The British Journal forthe Philosophy of Science, 17(4):319–321, 1967.

Ian Hacking. The Emergence of Probability: A Philosophical Studyof Early Ideas About Probability, Induction and Statistical Inference.Cambridge University Press, 2006.

bibliography 131

A. Hald, A. de Moivre, and B. McClintock. ’de mensura sortis’ or’on the measurement of chance’. International Statistical Review /Revue Internationale de Statistique, 52(3):229–262, 1986.

Marius Hofert and Mario V Wuthrich. Statistical Review of NuclearPower Accidents. SSRN Electronic Journal, 2011.

Geert Hofstede, Michael Minkov, and Gert Jan Hofstede. Culturesand Organizations: Software of the Mind. McGraw-Hill Education,third edition, 2010.

T Ishimatsu, N Leveson, J Thomas, M Katahira, Y Miyamoto,and H Nakao. Modelling and Hazard Analysis Using STPA. InConference of the International Association for the Advancement of SpaceSafety, Huntsville, 2010.

P Jones. Goldsboro Revisited or: How I learned to Mistrust theH-Bomb or: To Set the Record Straight. SFRD Rough Draft RS1651/058, Redacted Copy, DoE, October 1980.

Dan M kahan, Donald Braman, John Gastil, Paul Slovic, and C KMertz. Culture and Identity-Protective Cognition: Explaining theWhite-Male Effect in Risk Perception. Journal of Empirical LegalStudies, 4(3):465–505, November 2007.

R G Kasper. Perceptions of Risk and Their Effects on DecisionMaking. In R C Schwing and Albers W A, editors, Societal RiskAssessment, pages 71–84. Boston, 1980.

J. L. Kelly. A new interpretation of information rate. The Bell SystemTechnical Journal, 35(4):917–926, July 1956. ISSN 0005-8580. doi:10.1002/j.1538-7305.1956.tb03809.x.

David King, Ye Qi, Dadi Zhou, and Arunabha Ghosh. ClimateChange: A Risk Assessment. Technical report, Foreign andCommonwealth Office, July 2015.

Frank H Knight. Risk, Uncertainty, and Profit. Houghton MifflinCompany, Boston, 1921.

Robert J Lempert and Myles T Collins. Managing the Risk ofUncertain Threshold Responses: Comparison of Robust, Optimum,and Precautionary Approaches. Risk Analysis, 27(4):1009–1026,August 2007.

Bev Littlewood and Lorenzo Strigini. Validation of Ultra-HighDependability for Software-based Systems. Springer BerlinHeidelberg, Berlin, Heidelberg, 1995.

J P Martino. An Introduction to Technological Forecasting. Taylor &Francis Inc., 1973.

M G Morgan, M Henrion, and M Small. Uncertainty. A Guide toDealing with Uncertainty in Quantitative Risk and Policy Analysis.Cambridge University Press, Cambridge, 1998.


Béla Nagy, J Doyne Farmer, Jessika E Trancik, and John PaulGonzales. Superexponential long-term trends in informationtechnology. Technological Forecasting and Social Change, 78(8):1356–1364, 2011.

NTSB. Auxiliary Power Unit Battery Fire, Japan Airlines Boeing787-8, JA829J, Boston, Massachusetts, January 7, 2013. AccidentInvestigation Report NTSB/AIR-14/01, National TransportationSafety Board, Washington, DC, 2014.

Fred B Oswald, Michael Savage, and Erwin V Zaretsky. Space Shut-tle Rudder/Speed Brake Actuator—A Case Study. ProbabilisticFatigue Life and Reliability Analysis. Tribology Transactions, 58(1):186–196, 2014.

Wendy S Parker and James S Risbey. False precision, surprise andimproved uncertainty assessment. Phil. Trans. R. Soc. A, 373:1–13,December 2015.

O. Peters and M. Gell-Mann. Evaluating gambles using dynamics.Chaos: An Interdisciplinary Journal of Nonlinear Science, 26(2):023103,Feb 2016. ISSN 1089-7682. doi: 10.1063/1.4940236. URL http:

//dx.doi.org/10.1063/1.4940236.

Ole Peters. The time resolution of the st petersburg paradox.Philosophical Transactions of the Royal Society of London A: Math-ematical, Physical and Engineering Sciences, 369(1956):4913–4931,2011. ISSN 1364-503X. doi: 10.1098/rsta.2011.0065. URL http:

//rsta.royalsocietypublishing.org/content/369/1956/4913.

H. Petroski. To Engineer is Human, The Role of Failure in SuccessfulDesign. Random House, New York, First Vintage Edition edition,1992.

H.W.W Potts, J.E. Anderson, L. Colligan, P. Leach, S. Davis, andJ. Berman. Assessing the validity of prospective hazard analysismethods: a comparison of two techniques. BMC Health ServicesResearch, 14(1):41, 2014. ISSN 1472-6963. doi: 10.1186/1472-6963-14-41. URL http://dx.doi.org/10.1186/1472-6963-14-41.

M V Ramana. Beyond our imagination: Fukushima and theproblem of assessing risk. April 2011.

F Redmill. Exploring subjectivity in hazard analysis. EngineeringManagement Journal, 12(3):139–144, July 2001.

Amanda Ripley. The Unthinkable. Who survives when disasterstrikes - and why. Random House, London, 2008.

John Rushby. Modular Certification. Technical report, Menlo Park,July 2002.

Boeing Aviation Safety. Statistical summary of commercial jetairplane accidents (1956-2015). Statistical report, Boeing AircraftCompany, Seattle, Washington, 2015.

http://dx.doi.org/10.1063/1.4940236

http://dx.doi.org/10.1063/1.4940236



http://dx.doi.org/10.1186/1472-6963-14-41

bibliography 133

E M Schlosser. Goldsboro revisited. Technical report, SandiaLaboratories, October 1969.

David H Silvera, Frank R Kardes, Nigel Harvey, Maria L Cronley,and David C Houghton. Contextual Influences on OmissionNeglect in the Fault Tree Paradigm. Journal of Consumer Psychology,15(2):117–126, January 2005.

P Slovic and E U Weber. Perception of Risk Posed by ExtremeEvents. In Risk Management strategies in an Uncertain World, pages1–21, New York, 2002.

P Slovic, D Kahneman, and A Tvsersky. Judgment under uncertainty:Heuristics and biases. Cambridge University Press, New York, 1982.

Paul Slovic and Ellen Peters. Risk Perception and Affect. CurrentDirections in Psychological Science, 15(6):322–325, December 2006.

Didier Sornette. Dragon-Kings, Black Swans and the Prediction ofCrises. SSRN Electronic Journal, 2009.

J. Soukas. The limitations of safety and risk analysis. In Institute ofChemical Engineers Symposium, 11. Institute of Chemical Engineers,March 2007.

Joseph M Sussman. Ideas on Complexity in Systems - TwentyViews. Technical report, Massachusetts Institute of Technology,February 2000.

D Swallom. Mathematical Techniques to Improve the Utility ofa Hazard Risk Matrix. In th International System Safety Conference,pages 1–10. ISSS, 2011.

Nasim Taleb. The Black Swan. The Impact of the Highly Improbable.Random House, 2007.

W E Vesely and D M Rasmuson. Uncertainties in Nuclear Prob-abilistic Risk Analyses. Risk Analysis, 4(4):313–322, December1984.

Thomas S Wallsten and et al. Measuring the vague meanings ofprobability terms. Journal of Experimental Psychology: General, 115(4):348–365, 1986.

David Wood. Eight Theses reflecting on Stephen Toulmin. In DavidHitchcock and Bart Verheij, editors, Arguing on the Toulmin Model,pages 379–398. Springer Netherlands, Dordrecht, 2006.

Brian Wynne. Unruly Technology: Practical Rules, ImpracticalDiscourses and Public Understanding. Social Studies of Science, 18

(1):147–167, June 2016.

Pamela Zave and Michael Jackson. Four dark corners of require-ments engineering. ACM Transactions on Software Engineering andMethodology (TOSEM), 6(1):1–30, January 1997.

Glossary

abstraction Abstractions are developed for some purpose, likemodels they have agency. Abstractions share all their featureswith what is being described by the abstraction, the reverse isnot true). Abstractions describe what they are abstractions of:every feature of the abstraction is faithfully reflected in the target;every true sentence concerning the abstraction is similarly true ofthe target, under the appropriate interpretation of the predicatesand names. 25

affect Affect is the faint whisper of emotion which in turn informsour perception of risk. Affect is not a response to evidencerather it influences our decisions directly. Slovic et al. formallydefine the term as meaning a) a specific quality of “goodness”or “badness” experienced as a feeling state (with or withoutconsciousness) and b) demarcating the positive or negativenature of a stimulus. This can lead to problematic responses todecisions about risks, for example probability neglect. 46, 50–52

aleatory ‘Aleatory’ derives from the Latin word for game of chance.64

aleatory uncertainty Aleatory uncertainty characterises the inherentrandomness in the behaviour of the system under study andis therefore irreducible. Aleatory uncertainty may formallycharacterised using mathematics i.e pascalian calculus. 64

allocative efficiency A state of the economy in which productionrepresents consumer preferences; in particular, every good orservice is produced up to the point where the last unit providesa marginal benefit to consumers equal to the marginal cost ofproducing. 87

analysis Analysis is the reductionist process of breaking a complextopic or substance into smaller parts in order to gain a betterunderstanding of it. However Bertalanffy points out for a reduc-tionist analysis process to work it requires that the relationshipsbetween the ‘parts’ be weak and that their interactions be linear,or at least reasonably approximated as linear. Given that systemsdefinitionally do not satisfy these properties then terms such as‘system analysis’ as far as they embody reductionist techniquesare inherently self contradictory. 25, 50, 64, 71, 107, 108


assumption An assumption is something that is taken for grantedor presupposed to be the case without direct proof, they are ineffect beliefs that we act upon as if they were true. Assumptionsmay be explicit or implicit. Implicit (unstated) assumptions arepotential sources of ontological uncertainty and risks. The firstrisk is simply that the assumption is wrong, the second andmore subtle, is that assumptions are often made in regard to aspecific context, if you change the context then the assumptionmay no longer hold true. Assumptions in turn guide and directwhat we infer about a situation or set of data. 18, 71, 74

binomial Binomial (bi=two) indicates two possible outcomes, e.g.head or tails, pass or fail and failure or success. 29

byzantine failure The loss of a system service due to a fault thatpresents different symptoms to different observers (a Byzantinefault). For a system to exhibit a Byzantine failure, there must bea system-level requirement for consensus. If there is no consen-sus requirement, a Byzantine fault will not result in a Byzantinefailure (Kevin Driscoll). 110, 114, 115

cognitive bias The concept of cognitive biases was posited by Tver-sky and Kahneman in 1972 as a consequence of their work onthe problem of inumeracy or the inability of non-experts to reasonintuitively with greater order of magnitude numbers. They ex-plain this as a result of heuristics which simplify reasoning butcan lead to errors. Some biases are particular to groups, such asrisky shift, while others are individual in nature. Some affect de-cision making, such as the sunk cost fallacy while others affect thejudgement of probability, such as illusory correlation, while othersmay affect an individual’s motivation. This is an area of ongoingresearch and the number of identified biases are continuallybeing added to. 48

complexity The stem of the word complexity i.e. complex is com-posed of the latin words com ‘together’ and plex ‘woven’. Com-plexity & the related concept of simplicity are observer depen-dent system properties. Complexity as such is a much debatedphilosophical concept but some indicators of complexity in asystem are the existence of:

• emergent behaviour (cascading or compositional).

• observer dictated boundaries.

• openness (interaction with the environment).

• history dependence.

• hierarchical structure.

• small world effects.

• nonlinearity.

glossary 137

• stochastic behaviour.

• feedback loops

. 71, 76

corrigibility Capable of being corrected, reformed, or improved.A decision is easy to correct, or highly corrigible, when, if it ismistaken, the mistake can be discovered quickly and cheaplyand when the mistake imposes only small costs which can beeliminated quickly and at little expense (David Collingridge).Implicit in corrigibility is the reversibility of a decision or action.100–102

culture A concept originating in the anthropological studies ofthe mid-twentieth century culture and can be though of as the‘collective programming’ of the human mind with a set of valuesand practices which distinguishes the members of one group,or category of people, from another. As such culture is learnedrather than inherited and can be though of as sitting between anindividual’s personality traits on the one hand and the universalsof human nature on the other. Culture exists and interacts atmany levels, that is we can look at culture in terms of national,regional, gender, age or organisational differences in behaviours.45, 46

design hypothesis A design hypothesis is a prediction that a specificdesign will result in a specific outcome. Normatively a designhypothesis should: identify it’s provenance, e.g the theory, prac-tice or standards from it is derived, provide a concise descriptionof the design, state what the design must achieve in a verifiablefashion, and clearly identify critical assumptions that support it.73, 75

dragon king Corresponding to meaningful outliers, which are foundto coexist with power laws in the distributions of event sizesunder a broad range of conditions in a large variety of systems.The presence of such events indicates that there is some thepossibility for a transient organisation of the system into extremeevents that are statistically and mechanistically different from therest of their smaller siblings. 32, 33

epistemic From the greek ‘episteme’, meaning ‘knowledge’. Thebranch of philosophy called epistemology studies the nature ofknowledge, justification, and the rationality of belief. 64

epistemic uncertainty Epistemic uncertainty reflects our lack ofknowledge about a specific fact that we could in principal know,but do not. 64

evidence Most broadly evidence can be anything that is presentedin support of some assertion. From a scientific perspective that


something is usually empirical in nature and intended to eithersupport or falsify a hypothesis. In the context of system safetyevidence is intended to support or falsify some safety hypothesisabout the system. 46, 47, 50, 51, 63, 66

evidential weight Evidence weight has two different and distinctmeanings, firstly (after Nance) the degree to which a rationaldecision-maker is convinced of the truth of a proposition ascompared to some competing hypothesis (which could be simplythat the proposition is false), or (according to Keynes) a balancenot between the favourable and unfavourable evidence, but be-tween the absolute amounts of relevant knowledge and relevantignorance. 104

expectation Expectation is the likelihood weighted average of thepossible outcomes. In the case of risk it is the weighted averageof possible losses. 16, 18, 31, 32, 36, 37

experimenter’s regress Refers to the dependence between theoryand evidence. In order to judge whether evidence supportsa theory we must already have an expectation of our resultsbased on theory. Experimenter’s regress occurs when there isboth a phenomenon that the experimenter is unsure of and anexperimental procedure that the experimenter is unsure actuallyworks. 69, 70, 72

gross disproportion Gross disproportion is the decision criteria usedwhen determining what is reasonably practicable to do in relationto a risk to human safety. The term gross was deliberately chosento bias the decision maker towards action. Economic hardshipis not generally recognised by the courts in determining grossdisproportion. There is however no set or established value forwhat may be considered grossly disproportionate, that valuejudgement is left to the decision maker in the first instance. 93

hazard A hazard is an intermediate requirement that links a safetygoal to a satisficing set of safety constraints via an argument overcausality. To be complete it must include the rationale as to howthe goal is satisficed in the context of the risk domain. We useour control of hazards to assert that safety (a negative propertyof the system) is true via a proof by contradiction argument. 13,27

heuristic From the ancient greek to ‘find’ or ‘discover’. heuristicsare from a psychology perspective, simple efficient rules, learnedor hard-coded by evolutionary processes that explain how peoplemake decisions, judgments or solve problems when faced withcomplex situations, incomplete information or time compression.For example the availability heuristic uses readiness of recall asan analog for frequency when assessing probability, however if

glossary 139

events are under reported their relative probability will likewisebe estimated as low (a bias). 34, 46, 47, 109

hypothesis A tentative explanation usually in the form of seriesof propositions set as an explanation for an observation, phe-nomenon or problem that can be tested by further investigation.71, 75

ideal of mechanical objectivity As defined by John Downer the idealof mechanical objectivity is the idea that complex technologicalproperties like risk and reliability are wholly, objectively, andquantitatively knowable through a set of formalised rules andmethodologies. This ideal is used to portray risk assessments fortechnologies with catastrophic consequences as being inherentlyobjective, rational and therefore the results of such assessmentsas being highly trustworthy. 83

inductive reasoning Inductive reasoning, as used in science, logicand philosophy, means inferring general rules or laws fromobservations of particular instances. In effect it is the belief orassumption that unobserved cases will resemble observed cases.This usefully allows us to form theories about how things willbehave in the future based on observations of how similar thingsbehaved in the past. Safety analysis techniques such as FTAsor FMECAs are inherently exercises in inductive reasoning,although in the case of FTAs deductive (i.e. logical) operationsare used to derive cut sets of causal factors. 110

knowledge claim Something that the claimant believes to be true,yet is also open to discussion and debate. A knowledge claimis something that we believe we know and that we want toassess the validity of. There are two different types of knowledgeclaims-the first type are claims about the world and the secondtype are claims about knowledge itself, such as the degree ofconfidence we may have in evidence. 69, 76

model Models are developed for some purpose, like abstractionsthey have agency. Models describe what is being modelled onlyin certain ways: true sentences about the model may not be trueof the target, but the salient ones are. A model and what is beingmodelled should share some common abstraction. 16, 17, 25, 67,108

omission neglect bias A general insensitivity to missing informationwhich might be characterised as “what you pay attention willdictate what you’ll miss”. This results in the tendency of anobserver to place too much weight on the evidence that is visibleto them while under-estimating or overlooking the effect offactors that are not directly visible. 47, 68


ontological uncertainty Ontological uncertainty represents a stateof complete ignorance. Not only do we not know, but we don’teven know what we don’t know. We cannot readily access therequisite knowledge because we simply don’t know where tolook in the first instance. 64

ontology The philosophical study of the nature of being or existence.73

operationalism Operationalism is the philosophy developed byPercy Bridgman in his 1927 work The Logic of Modern Physicsthat what we mean by any concept is nothing more than a set ofoperations. The concept is synonymous with the correspondingset of operations, no more and no less. Operational definitionsbecome useful when a concept cannot be directly measured butmay be inferred from the presence of other phenomena, as is thecase with risk. Bridgman’s concept of operationalism has muchin common with the logical positivism of the Vienna Circle whoin turn trace their ideas back to the empiricism of Hume. 18, 19,21

outlier An outlier is an observation that lies to be an abnormaldistance from other values in a random sample from a popu-lation. In classical statistics such observations are treated asanomalies that should be discarded, because not to do so invali-dates the idea of a population statistic as a meaningful measure,one can after all still drown in a river that is three feet in depth.In contrast theories such as Extreme Value and Dragon Kingsemphasise the importance of understanding such ‘beyond thenormal’ events. 32

pascalian probability ‘Pascalian’ or aleatory probability is the conceptof probability captured by the axioms and theorems of prob-ability calculus. The name commemorates Blais Pascal as theprincipal father of the calculus. 17, 30, 46, 75

precautionary principle First articulated at the UN Conference in Rio1992, the principle states that where there are threats of seriousor irreversible environmental damage, lack of full scientificcertainty shall not be used as a reason for postponing preventivemeasures to prevent degradation. 99

principal agent dilemma The principal-agent dilemma or problemoccurs when one person or entity, the ‘agent’ is able to make de-cisions on behalf of, or that impact, another person or entity, the‘principal’. The root of the problem is the asymmetric nature ofinformation and the possibility that the agent will act to furthertheir interests rather than their principal. Many regulations areintended to address this problem. 85

probability The word probability derives from the latin probabilitas,which also means ‘probity’, that is a measure of the authority of

glossary 141

expert opinion and therefore how much weight we should placeupon their evidence. However this differs from the pascaliandefinition of probability which is derived from inductive rea-soning and statistical inference. Modern probability theory canbe further divided into two broad camps that of the frequentists,who believe that probability is strictly derived from observed setsof events, and that of the subjectivists or Bayesians, who believethat probability reflects an observer’s degree of belief given all thefacts in their possession. 15–19, 37–39

prospect theory Prospect theory is a behavioural economic theorydeveloped by Daniel Kahneman and Amos Tversky in the early1990s to describe the way people actually choose between riskyalternatives where the probabilities are known. Based on em-pirical evidence the theory shows that a loss is more significantthan an equivalent gain, that a sure gain (i.e the certainty andpseudo-certainty effect) is favoured over a probabilistic gain, andthat a probabilistic loss is preferred to a definite loss. 52

reasonably practicable Reasonably practicable is a legal term en-shrined in UK case law since the case of Edwards v. NationalCoal Board in 1949. There are two parts to what is ‘reasonablypracticable’. First the decision maker must consider what canpracticably be done in the circumstances. Note that the termpracticable has a narrower meaning than the term practical, tobe practicable a measure must not only be possible and use-ful (practical) but also feasible in the circumstances. Then thedecision maker must consider whether it is reasonable, in thecircumstances to do all that is possible. This means that whatcan be done should be done unless it is reasonable in the circum-stances for the duty-holder to do something less. The ruling ofthe Edwards case was that in determining what was reasonablea risk must be significant in relation to the sacrifice (in terms ofmoney, time or trouble) required to avert it. Risks must thereforebe eliminated or minimised unless there is a gross disproportionbetween the costs and benefits of doing so. 92

reductionist A reductionist technique is one which follows a processof describing a phenomena in terms of more ‘primitive’ phenom-ena to which the original is considered to be equivalent. Classicalsystem safety analyses are invariably reductionist in nature. 110

risk The word risk derives from an early Italian word ‘risicare’,which means ‘to dare’. In this sense risk is a choice rather thana fate and with that authority of choice goes (at least in a justsociety) some measure of responsibility and accountability.The other aspect of risk is that there is both an upside and adownside to the decision to take a risk. We may risk some lossbut this is always balanced against the potential gain. In the17th century this concept of risk became synomous with the


mathematical concept of ‘expectation of loss’ through the worksof de Moivre. 15, 16

risk perception The subjective aspects of risk evaluation. The idea ofrisk perception arose from the observation lay persons assess-ments of risk often differed from the risk assessment of experts.43

safety analysis When ‘analysis’ is used in the context of ‘safetyanalysis’ it is generally taken to mean that we are determiningthe hazardous consequences of failures or alternatively thecausal factors of accidents. Safety analysis are predominantlyreductionist techniques see Fault Tree Analysis (FTA) and FailureModes and Effects Criticality Analysis (FMECA) which argueinductively from consequence to causal factors and causal factorto consequence respectively. 110

satisficing Satisficing is a decision-making strategy, or heuristic,that entails searching through a set of available alternatives untilan acceptability threshold is met (H.A.Simon). In Simon’s viewmost real world problems are either computationally intractabil-ity or missing information so that determining a mathematicallyoptimum solution is impossible. Thus decision makers can satis-fice either by finding optimum solutions for a simplified world,or by finding satisfactory solutions for a more realistic world. 52,100

surprise index A standard measure of overconfidence. This indexmeasure what percentage of the true measured value of a pa-rameter lie outside an assessor’s 98% confidence interval. Forexample NASA’s quantitative estimates of the expected lossrate for shuttle failure predicted probabilities as low as 1 loss in100,000 flights. The actual loss rate experienced was 2 losses in135 flights. 76

system After Bertalanffy a system is defined as a complex of compo-nents in interaction, or elements in a standing relationship. Weare usually interested in teleological (purposeful) systems. 11, 13,135

uncertainty In general, uncertainty can be defined as having limitedknowledge about a subject. However uncertainty is not simplythe absence of knowledge, it can result from the inadequacy ofthe information that we possess, which may be considered tobe inexact, unreliable or in conflict (including the opinions ofexperts). 20, 32

value of a statistical life In economic terms the Value of a StatisticalLife (VSL) is the amount of money a person (or society) is willing

glossary 143

to spend to save a life. The term ’statistical’ is introduced be-cause these values are reported in units that match the aggregatedollar amount that a large group of people would be willing topay for a reduction in their individual risks of dying in a year,that would result in one less death among the group during thatyear on average. 90

values Values are broad tendencies to prefer certain states of affairsto others. Values are feelings with an arrow to it they have aplus and a minus side, for example good vs evil, rationality vsirrationality. Values are learned very early in childhood and areacquired implicitly rather than consciously. As a result manyvalues are unconscious to those who hold them and we generallycan only infer them from people’s actions. 82

Acronyms

AEA Action Error Analysis. 26

CIL Critical Items List. 61

FMEA Failure Modes and Effects Analysis. 61

FTA Fault Tree Analysis. 17, 27

HAZOPS Hazards and Operability Study. 26, 27

NAA North American Aircraft. 13

NASA National Aeronautics and Space Administration. 12–14

NPP Nuclear Power Plant. 25

NRC Nuclear Regulatory Commission. 26

PRA Probabilistic Risk Assessment. 17, 25, 63, 67, 103

STAMP System Theoretic Accident Model and Processes. 27

TLE Top Level Event. 23

WSA Work Safety Assessment. 26

I wouldn't do that if I were you - Critical Uncertainties...know there are some things we do not...

Documents

Transcript of I wouldn't do that if I were you - Critical Uncertainties...know there are some things we do not...