A Bayesian method for using simulator data to enhance human errorprobabilities assigned by existing HRA methods

Katrina M. Groth a,n, Curtis L. Smith b, Laura P. Swiler a

a Sandia National Laboratories, Albuquerque, NM 87185-0748, USAb Idaho National Laboratory, Idaho Falls, ID 83415-3850, USA

a r t i c l e i n f o

Article history:Received 2 April 2013Received in revised form13 February 2014Accepted 31 March 2014Available online 5 April 2014

Keywords:Human reliability analysis (HRA)Bayesian inferenceSimulator dataNuclear power plantHuman performance data

a b s t r a c t

In the past several years, several international organizations have begun to collect data on humanperformance in nuclear power plant simulators. The data collected provide a valuable opportunity toimprove human reliability analysis (HRA), but these improvements will not be realized withoutimplementation of Bayesian methods. Bayesian methods are widely used to incorporate sparse datainto models in many parts of probabilistic risk assessment (PRA), but Bayesian methods have not beenadopted by the HRA community. In this paper, we provide a Bayesian methodology to formally usesimulator data to refine the human error probabilities (HEPs) assigned by existing HRA methods. Wedemonstrate the methodology with a case study, wherein we use simulator data from the HaldenReactor Project to update the probability assignments from the SPAR-H method. The case studydemonstrates the ability to use performance data, even sparse data, to improve existing HRA methods.Furthermore, this paper also serves as a demonstration of the value of Bayesian methods to improve thetechnical basis of HRA.

& 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Probabilistic risk assessment (PRA) is a conceptual and compu-tational framework for supporting regulatory and operationaldecision-making in the nuclear power industry. Within this frame-work, operational experience and data play an important role inthe assignment of probabilities. To ensure that these decisionsare robust and defensible, it is desirable to maximize the use ofempirical data to develop and validate the probabilistic modelsused in PRA. Because of PRA's focus on low-frequency scenarios inhigh reliability systems, empirical data are often sparse. Conse-quently, Bayesian methods are widely used in PRA for formallyassigning probabilities using both subjective information andobjective data [1–4]. However, the Bayesian approach was notimplemented during development of many widely used HRAmethods [5].

Due to scarcity of human performance data, many HumanReliability Analysis (HRA) models used in PRA have been built onexpert information. There have been repeated calls to expand thetechnical basis of HRA by systematically integrating informationfrom different domains [6–11]. Bayesian methods, which offer the

framework to formally combine expert information and empiricaldata, can be used to address these calls.

Recent activities by many international agencies have resultedin several promising simulator data-collection projects (e.g, [12–16]).Some of these activities are focused on developing objective fre-quencies (e.g., [17]), but as Podofillini and Dang note, none of thesesources allows obtaining statistically significant information neededto produce a comprehensive HRA model [5].

Conceptually, Bayesian methods can be applied to a wide rangeof HRA problems. The feasibility of implementing Bayesian methodsto enhance HRA was explored as an academic activity in NUREG/CR-6949 [18]. NUREG/CR-6949 has been criticized as too general,and reviewers requested more targeted applications of Bayesianmethods to specific problems [18, p. 59–60].

Applying Bayesian methods computationally requires a moredetailed look at specific problem spaces. One area of Bayesianmethods has been gaining popularity in the HRA community: theuse of Bayesian Networks (BNs, also called causal models orBayesian Belief Networks, BBNs). Many researchers [19–29] areusing BNs to enhance HRA. However, BNs are only one imple-mentation of the larger world of Bayesian methods that canbenefit HRA. There remains a full range of Bayesian methods andassociated computational techniques that can be applied withinHRA, but which the HRA community has not adopted. This use ofadditional Bayesian methods can apply outside of the BN

Contents lists available at ScienceDirect

journal homepage: www.elsevier.com/locate/ress

Reliability Engineering and System Safety

http://dx.doi.org/10.1016/j.ress.2014.03.0100951-8320/& 2014 Elsevier Ltd. All rights reserved.

n Corresponding author. Tel.: 1 (505) 844 6766.E-mail address: kgroth@sandia.gov (K.M. Groth).

Reliability Engineering and System Safety 128 (2014) 32–40

framework, as is done for hardware PRA, and can also becomplementary to the use of Bayesian Networks (BNs) in HRA.

This paper proposes the use of Bayesian methods to formallyincorporate simulator data into estimation of human errorprobabilities (HEPs) in existing HRA methods. The approachenables even limited amounts of simulator data to be used toenhance the technical basis of existing and future HRA methods.Ideally, the approach demonstrated in this paper would be usedalong with the approach for formally incorporating expert esti-mates in [5] to maximize the information basis used to developHRA models.

We demonstrate the methodology by using simulator data fromthe Halden Reactor Project [14] to update the SPAR-H [30] HRAmethod. The results discussed in this paper are specific to thiscombination of information, but the methodology provided is applic-able to any combination of HRA methods and data. The intention istwofold: to provide a framework for formally enhancing existingHRA methods with new simulator data, and to demonstrate that itis possible to use sparse data from different sources to enhance thetechnical basis of HRA.

In Section 2, we draw parallels between hardware PRA andHRA to illustrate that Bayesian updating can be applied to para-meters of HRA models in the same way it is used to update analeatory model with hardware data. In Section 3, we provide anintroduction to the Bayesian inference process. In Section 4, wepresent the methodology for using simulator data to update beliefsabout human error probability. In Section 5, we present a casestudy where we used data extracted from Halden simulatorexperiments to update HEPs from the SPAR-H method. In Section6, we provide a discussion of the work, and we present finalconclusions in Section 7.

2. The meaning of probability in PRA and HRA

PRA uses a combination of models to predict system behaviorwith possible consequences and to assign probabilities to differentcontributors and outcomes. Probability is used as a tool todistinguish more likely scenarios from less likely scenarios. Thescenarios are decomposed into a set of basic events (e.g., compo-nent failure on demand and human errors), which are encoded indeterministic logic-based models (e.g., Fault Trees and EventTrees). A probability is associated with each basic event, and thebasic event probabilities are combined according to the logicmodel to provide scenario probabilities for use in decision making.

An example basic event in PRA is “failure of valve in a closedposition.” The probability of each basic event is expressed by aprobability model and an associated set of parameters. Eachprobability model could be a common probability distribution,such as the Bernoulli distribution with parameter p or thebinomial distribution with parameters p and n. Each could alsobe a more complicated probability model such as a BN, which isassociated with a set of parameters for each causal factor in themodel. For either modeling approach the analyst assigns modelparameters, which represent his/her current state of knowledgebased on all of the available information. These models are derivedfrom either (a) subjective information (e.g., about the chance ofnot performing an intended function), (b) actual failure data, or acombination of (a) and (b).

The goal of quantitative HRA is to assign an HEP, where an HEPis the probability of a human failure event, HFE. Example HFEs are“failure to initiate Feed and Bleed” and “failure to align electricalbus to alternative feed.” In probability language, the HEP is anassignment of the belief of the truth of the proposition “the humanresponse to event X will not satisfy system requirement Y.” Byreplacing the word “human” in the previous sentence with the

word “hardware,” it becomes apparent that assigning the HEPwith an HRA model is not conceptually different than assigning ahardware failure probability with a common probability distribu-tion such as the binomial distribution.

A major difference between HRA and hardware PRA is in theuse of context in the models; HRA methods are to assignprobabilities for different contexts, whereas for hardware, contextis often ignored and statistical data are used in combination withsubjective information to assign probabilities. There are numerousHRA methods available that provide models for assigning the HEPbased on the context of the performance. That is, the HRA methodsprovide the conditional probability of an HFE, given the context ofperformance PðHFEjcontextÞ. In many HRA methods, the context isrepresented by Performance Shaping Factors (PSFs) or Perfor-mance Influencing Factors (PIFs); some newer HRA methods alsoinclude plant conditions and cognitive factors in the discussion ofcontext. These context factors are discretized into levels or states.

The use of detailed HRA models to predict outcomes for humanevents is no different than the use of a binomial distribution topredict outcomes for hardware events. The only differencebetween the two cases is the number of parameters included inthe model used to make the prediction.

3. Overview of Bayesian inference

The goal of applying Bayesian inference is to reason about a setof hypotheses. In other words, we will determine which hypoth-eses are logically more plausible, given the initial knowledge andthe evidence/data available. The hypothesis, H, expresses a belief,e.g., about an event or a set of outcomes or about the parametersof a model. The data, D, represent observations relevant to thehypothesis.

The heart of the Bayesian inference process is Bayes' Theorem(Eq. (1)), which provides the conceptual and mathematical meansfor combining different information in the context of a probabil-istic model:

PðHjDÞ ¼ PðHÞPðDjHÞPðDÞ ð1Þ

We apply Bayes' Theorem to obtain the posterior distribution,PðHjDÞ, which expresses the plausibility of the hypothesis, giventhe data. We follow this process to determine the probability thata hypothesis is true, conditional on all available evidence. Bayes'Theorem can be applied iteratively to incorporate a variety of newevidence to make inferences about the same hypothesis. As such,the posterior probability from one analysis becomes the priorprobability for the next. This process allows us to coherentlyincorporate a variety of information sources into a single structurein order to make inferences.

To apply Bayes' Theorem, information known about a hypoth-esis (independent of the data) is described with a prior; this prioris encoded via a probability model, P(H). The prior is a probabilitydistribution which expresses the plausibility of the hypothesis;this prior represents the analyst's initial knowledge about thehypothesis.1

The next step is to collect or obtain data, D, and express thosedata in a format that can be used to update the prior distribution.To do this, we use a likelihood model. The likelihood model allowsus to translate the observable data into probabilistic information.The likelihood model represents the process providing the data;

1 To reiterate, the Bayes' prior is intended to capture a state of knowledgeindependent of data collection. This does not imply that the event we are modelinghas this probability distribution as an inherent property; rather, the probabilitydistribution expresses beliefs about the event.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–40 33

in other words, it is a model of performance. This likelihoodexpresses the probability of the data, given the truth of thehypothesis, PðDjHÞ. In Eq. (1) this likelihood is normalized by theprobability of the data over all specified hypotheses, P(D). For khypotheses, P(D) can be calculated as PðDÞ ¼∑k

i ¼ 1PðHiÞPðDjHiÞ.Different likelihood models will be appropriate for different typesof data and hypotheses.

Bayes' Theorem provides the mathematical framework forassessing the posterior distribution, but executing the calculationscan be an involved process. For discrete probability distributions,the posterior distribution can be easily calculated using Eq. (1).However, for continuous probability distributions, the calculationof P(D) involves integration, as shown in the continuous form ofBayes' Theorem:

π1ðθjxÞ ¼f ðxjθÞπ0ðθÞ

f ðxÞ ¼ f ðxjθÞπ0ðθÞRf ðxjθÞπ0ðθÞ dθ

ð2Þ

This integration cannot be obtained in a closed form for many formsof the likelihood function. In these cases, sampling methods such asMarkov Chain Monte Carlo Methods (MCMC) can be used to solvefor posterior distributions. MCMC sampling can be implemented inprograms such as OpenBUGS (formerly WinBUGS) or Matlab. Forsome combinations of likelihood functions and prior distributionsthat are conjugates, the integration has a closed form. For theseconjugate distributions, the prior and the posterior distribution areof the same functional type, and the posterior distribution can becalculated directly. Many of likelihood functions commonly used inPRA have conjugate prior distributions [4].

3.1. Example application in PRA

In PRA, analysts are tasked with expressing a probabilisticmodel for each basic event. One of the most common types ofbasic event in PRA is “component fails on demand.” For theseevents and other Bernoulli processes, the most appropriate prob-ability model is the binomial distribution:

PrðX ¼ xÞ ¼ f ðxjpÞ ¼ n

x

� �pxð1�pÞn�x ð3Þ

The binomial distribution is used to describe uncertainty about thenumber of failures, x, that will occur in a given number ofindependent demands, n, given a parameter, p, which is inter-preted as the probability of failure-on-demand for the component.In PRA, this type of uncertainty is called aleatory uncertainty.The term aleatory refers to the stochastic or random nature of theoutcome of processes such as coin flipping or valve opening.The binomial distribution is used to express uncertainty aboutthe outcome of a series of events, given parameter p.

The parameter p is not directly observable, but it can beinferred from data or assigned by experts. In most PRA applica-tions the value of the parameter is also uncertain, due to sparsedata, partially relevant data, imprecision in the data or themethods, and so on. This type of uncertainty is denoted epistemicuncertainty. The term epistemic refers to lack of knowledge; inPRA, analysts are uncertain about the value of the parameters ofthe failure model. In PRA, the Bayesian inference process is used toexpress beliefs about the plausibility of different possible valuesfor the parameter (that is, to provide the plausibility of hypothesesof the form “the value of parameter p is z”). Effectively, Bayesianinference is used to provide a distribution for the parameters of afailure model.

Prior beliefs about the parameter can be specified using a varietyof distributions, depending on the information available. The betadistribution is commonly used as prior distribution for the para-meter, p, of the binomial distribution: πðpjα;βÞ � Betaðα;βÞ.Betaðα;βÞ indicates a beta random variable, which has probability

density function (pdf) given by the following equation:

f ðp;α;βÞ ¼ pα�1ð1�pÞβ�1

Bðα;βÞ ð4Þ

Note that the beta function, Bðα;βÞ, is used as a normalizationconstant in the beta distribution.

The beta distribution has the advantage of being conjugate tothe binomial distribution. This means that the posterior distribu-tion for this combination will also be a beta distribution, withparameters αpost and βpost. The values of these parameters, whichrepresent the combination of the prior and the data, are given bythe following equation:

αpost ¼ αpriorþx

βpost ¼ βpriorþn�x ð5Þ

Since the beta and binomial distributions are conjugate, theBayesian updating process is simple: the analyst sets the prior beliefabout p by assigning values to α and β. The data record x failures inn demands. The posterior distribution for p is a beta distributionwith parameters obtained from Eq. (5). This posterior modelexpresses epistemic uncertainty about the value of p, which is aparameter of the aleatory model (Eq. (3)) to be used in the PRA.To implement the posterior distribution within the aleatory model,several approaches could be used. Common approaches includesampling from the distribution for p and using either the moments(e.g., the mean) or the percentiles of the distribution for p.

4. Methodology to Bayesian update HEPs with simulator data

Once we understand that assigning HEPs is not conceptuallydifferent than assigning hardware event probabilities, the Bayesianinference process can be applied to update HEPs. Executing theBayesian inference process outlined in Section 3 requires severalsteps:

1. Define the hypothesis.2. Identify suitable sources of information.3. Specify the prior distribution.4. Specify the likelihood function.5. Conduct Bayesian updating to obtain a posterior.

In this section, we demonstrate how to conduct the Bayesianupdating process on HEPs generated by existing HRA methods usingsimulator data. The prior distribution is based on a current HRAmethod, and the likelihood function will be specified to matchsimulator data. It should also be noted that we could combine twoor more priors to have a mixture prior representing multiple methods,however this discussion is beyond the scope of this paper.

4.1. Step 1: define the hypothesis

The goal of HRA is to assign an HEP for the HFE, X. In PRAlanguage, we want to assign a probability the basic event “failureof human response to event X,” which is assumed to be Bernoulliprocess. As in Section 3.1, we use the binomial distribution (withunknown parameter p and n¼1) as the aleatory model. Since weare uncertain about the value of p, we wish to use Bayesianinference to obtain a posterior probability distribution for p.Therefore, the goal of the Bayesian inference is to express a beliefabout the hypothesis “the HEP for event X is p.”

4.2. Step 2: identify suitable sources of information

The HRA method and the simulator data selected to be used inthe Bayesian updating process must each be compatible with the

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–4034

hypothesis described in Step 1. Any HRA method that directlyassigns human error probabilities is a candidate for use inBayesian inference on the hypothesis. This includes methods suchas ASEP [31], CBDT [32], CESA-Q [33], CREAM [34], HEART [35],KHRA [36], SLIM-MAUD [37], SPAR-H [30], THERP [38], and others.While the focus of the current paper is on HRA methods thatdirectly assign HEPs, the Bayesian updating process can be used toupdate expert judgments (such as those used to assign theprobabilities as part of methods like ATHEANA [39]), as documen-ted by Podofillini and Dang [5].

There are several international projects collecting humanperformance data in nuclear power plant simulators, includingprojects at the OECD Halden Reactor Project [13,14], KAERI [40,15],the U.S. NRC [16,17] and others [12,41]. These types of experimentstypically provide the types of observables required as part of theBayesian inference process demonstrated in Section 5. Each ofthese simulator data sources should be compatible with thehypothesis, although many of the data sources are not yet avail-able to the HRA community. Data availability remains a key issuein HRA [9,42], so the choice of simulator data depends largely onavailability.

In addition to being compatible with the hypothesis, the HRAmethod and the simulator data must be compatible with eachother. The most important aspect of compatibility is the definitionof the HFE. Event/task decomposition must be consistent betweenthe HRA method and the simulator data: they must both definethe human failure event in a similar way. The issue of eventdecomposition, along with comparison of event decompositionmethods for many widely used HRA methods, is discussed in detailin [8]. A second critical area for compatibility between the HRAmethod and the simulator data source relates to the variables usedto define the context (e.g., PSFs and PIFs). The context variables inthe data source must be defined in such a way that enables themto be mapped onto the variables in the HRA method. Alternatively,both the data and the HRA method can be mapped to the generic,comprehensive set of PIFs developed to enable data analysis [24].

4.3. Step 3: specify the prior distribution

The prior distribution encodes information related to ourknowledge about the hypothesis identified in Step 1. The form ofthe prior distribution depends on the quality of the prior informa-tion. When prior information is absent, it is desirable to use a non-informative prior to represent ignorance. When more informationis available, informative prior distributions can be specified.

Existing HRA methods provide information, which can be usedto express the prior distribution on p, which is denoted p0. MostHRA methods provide a point estimate, or a formula to obtain apoint estimate, of the value of p for each context or combination ofPSFs. In SPAR-H, this point estimate can be interpreted as themean value of p0, Eðp0Þ, for a given context, as recommended by[30]. Note that other HRA methods may suggest that the pointestimate is a median or mode; typically these are close to themean value, especially when considering the overall uncertaintyfound in current HRA models. To implement this in a probabilitydistribution we use a limited information prior, based on theconstrained non-informative (CNI) distribution, where the con-straint is on the mean value of the distribution [43]. The CNIdistribution can be approximated by a Beta(α, β) distribution, withα¼0.5, and β derived from the constraint on the mean2 given by

the following equation:

EðBetaðα;βÞÞ ¼ ααþβ

¼ Eðp0Þ ð6Þ

In PRA, α can be thought of as the number of failures contained inthe prior distribution, and the sum (αþβ) can be thought of as thenumber of demands over which these failures occurred. Thereforethe CNI prior represents half of a failure occurring in (αþβ)demands, where β is determined from the predictions of theHRA model. We use this procedure to develop a CNI prior for eachcontext represented in the HRA method.

4.4. Step 4: specify the likelihood function

The likelihood is a mathematical function representing theprocess providing the data. In a Bayesian analysis, the likelihood isconditioned on the data. Thus, the form of the likelihood mustcoincide with the type of data to be collected. For the hypothesisindicated in Step 1, we are reasoning about the likelihood ofobserving a human failure upon demand. For this problem, therelevant data are the number of human failures, x, in a givennumber of independent opportunities for human failure, n. Thebinomial likelihood function is appropriate for this type of data.One assumption behind the binomial model is that the probability,p, does not change from one demand to another. Consequently,cases where a single individual or crew were repeatedly subjectedto the same scenario (resulting in learning) may need to be treatedwith a more complicated model, or the data should be filtered torepresent the most applicable case.

This likelihood model does not explicitly factor in informationabout the context of the performance (i.e., the PSFs). Rather, adifferent likelihood model is developed for each combination ofPSFs in the HRA method.

4.5. Step 5: conduct Bayesian updating to obtain a posterior

In the previous four steps, we established a hypothesis,gathered data relevant to the hypothesis, expressed prior beliefsabout the hypothesis, and identified the appropriate likelihoodmodel. In this step, we implement Bayes' Theorem to quantify theposterior belief about the hypothesis. In other words, we willdetermine the plausibility of the hypothesis given the prior andthe data.

Since we used a conjugate prior distribution and likelihoodfunction, the Bayesian updating process is straightforward. Theposterior distribution for p, denoted p1 is Beta(αpost, βpost), wherethe values of αpost and βpost are established using Eq. (5).

5. Case study: SPAR-H with Halden data

In this section, we implement the methodology detailed in Section4 to perform updating on the SPAR-H method using simulator datacollected by the Halden Reactor Project as part of the U.S. NuclearRegulatory Commission (NRC) HRA Empirical Study.

5.1. SPAR-H

The SPAR-H [30] human reliability analysis method was devel-oped to estimate HEPs for use in the SPAR nuclear power plantPRA models. SPAR-H is used as part of PRA in over 70 U.S. nuclearpower plants and by the event assessment programs at the NRC.

The SPAR-H method considers two plant states: at-power andlow power/shutdown, and two types of human activities: diag-nosis and action. The two types of activities use the sameequations and PSFs, but use different PSF multipliers and different

2 This relationship is used when Eðp0Þr0:5. For Eðp0ÞZ0:5, the values areswitched: β¼0.5 and α is derived from the mean. The interpretation of α and βremains the same.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–40 35

values for the nominal HEP (NHEP). In this paper, we present themodel for action tasks during at-power operations.

The SPAR-H method uses eight PSFs to represent the context.The first step in applying SPAR-H is to evaluate the level for eachPSF to determine the context and the associated HEP multipliers.Each PSF has three to five levels,3 shown in Table 1, and each levelis associated with a multiplier. Each possible combination of levelsfor the 8 PSFs forms a single context; for the eight PSFs, in SPAR-H,there are 19,440 possible contexts (combinations).

The second step is to calculate HEP using equations provided inthe worksheets. Two equations are provided; which equation isused depends on the number of negative PSFs (any PSF where theassigned level has a multiplier greater than 1). Eq. (7) is used tocalculate the HEP for situations with fewer than three negativePSFs. Eq. (8) is used if there are three or more negative PSFs:

HEP ¼NHEP �∏8

1Si ð7Þ

HEP ¼ NHEP �∏81Si

NHEP � ð∏81Si�1Þþ1

ð8Þ

where Si is the multiplier associated with the assigned level of PSFi. For diagnosis tasks NHEP¼0.01 and for action tasks NHEP¼0.001.The SPAR-H guidance suggests a lower limit of 1� 10�5 forthe HEP.

Since the SPAR-H model adjusts a nominal probability by adiscrete multiplier, the output of the model will be a set of discreteprobabilities that are a function of the specific PSF levels. Since thebinomial model we use for Bayesian inference has a parameter pthat can take on any value, it is able to represent the probabilityoutputs from the SPAR-H model.

5.2. Simulator data

5.2.1. Experimental conditionsIn 2010, a team of researchers from the NRC, the Halden

Reactor Project, Idaho National Laboratory, Sandia NationalLaboratories, and the Electric Power Research Institute conductedan experiment at the training simulator of a PWR plant in theUnited States. The experiment was designed to collect simulatordata to benchmark and improve existing HRA methods as part ofthe NRC's HRA Empirical Study activities [44,45]. The experiment,which is fully documented in [14], used the PWR simulator tochallenge four operating crews under a variety of different con-texts. In each trial, an operating crew was presented with one ofthree different simulator scenarios:

� Scenario 1 – Total loss of feedwater, followed immediately by aSteam Generator Tube Rupture (SGTR).

� Scenario 2 – Loss of component cooling water (CCW) andreactor coolant pump (RCP) seal water.

� Scenario 3 – A SGTR scenario (without complicating factors).

Scenarios 2 and 3 were designed to each test one HFE thatwould be used in PRA. Scenario 1 was designed to test two HFEsfor each crew. The criteria for occurrence of an HFE in thesimulator were

� HFE 1A – Failure to establish feed and bleed within 45 min ofthe reactor trip, given that the crews initiate a manual reactortrip before an automatic reactor trip.

� HFE 1C – Failure of crew to isolate the ruptured steam generatorand control pressure below the steam generator (SG) poweroperated relief valve (PORV) setpoint to avoid SG PORV open-ing, within a 40 min time frame.

� HFE 2 – Failure of the crews to trip the RCPs and start thePositive Displacement Pump (PDP) to prevent RCP seal Loss ofCoolant Accident (LOCA).

� HFE 3 – Failure of crew to isolate the ruptured steam generatorand control pressure below the SG PORV setpoint before SGPORV opening.

Four crews participated in the experiments. For three of thecrews the composition was one Shift Manager (SM), one UnitSupervisor (US), one Shift Technical Advisor (STA) and two ReactorOperators (RO). In one crew the SM was not present and insteadthey had three ROs. Three of the crews participated in each of thefour scenarios. A fourth crew only participated in three of thescenarios due to simulator malfunction. In total, the team collecteddata for 15 experimental trials.

5.2.2. Experimental dataMultiple types of information were gathered during and after

each of the scenarios. The data include second-by-second simu-lator logs, audio and video recording of the crew, observationsfrom the experimental team, and observations from plant trainingexperts. After the scenarios, the Halden experimental team con-ducted crew member interviews and distributed questionnaires toall crew members. The crew member questionnaires included anadapted version of the NRC Simulator Crew Evaluation Form,

Table 1SPAR-H PSFs, levels for each PSF, and multipliers for each level.

PSF PSF level Multiplier

Available time Expansive 0.01Extra 0.1Nominal 1Barely adequate 10Inadequate HEP¼1.0

Stressors Nominal 1High 2Extreme 5

Complexity Nominal 1Moderate 2High 5

Experience/training High 0.5Nominal 1Low 3

Procedures Nominal 1Avail., but poor 5Incomplete 20Not available 50

Ergonomics/HMI Good 0.5Nominal 1Poor 10Missing/misleading 50

Fitness for duty Nominal 1Degraded fitness 5Unfit HEP¼1.0

Work processes Good 0.5Nominal 1Poor 5

3 Note that the SPAR-H method also has an “Insufficient Information” level foreach PSF, with a corresponding multiplier of 1. This is not included in Table 1because Bayesian methods use prior information to enable inference when there ismissing information. See [27] for more information.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–4036

NUREG-SR1020 rev. 9, Form ES-604-2 [46]. Due to the sensitivenature of these data, only the experimental team had access to theraw experimental data.

The Halden team processed the data to remove all identifyingdetails about the crews before releasing any information to otherresearch teams. The team issued report HWR-981, [14], whichprovides detail summary data for each experimental trial. Thereport includes detailed descriptions of the scenarios, anonymizedcrew evaluations, and both the context and the outcome of eachexperiment. The context was described in natural language, andthe Halden team also provided ratings for eleven PSFs: stress,adequacy of time, team dynamics, work processes, communica-tion, scenario complexity, indications of conditions, human–machine interface, training and experience, procedural guidance,and execution complexity. The Halden team rated these PSFs on afour-point scale: nominal or positive, not a driver, negative driver,and main negative driver. While the Halden team did not directlycollect information about the PSFs used in a specific HRA method,the anonymized Halden data contains enough information to beused to extract PSFs for many methods.

5.3. Extracted data

The authors used the Halden report [14] and the anonymizedsimulator crew evaluation forms to assign a level for the SPAR-HPSFs for each of the 15 experimental trials. Expert judgment andqualitative information in [14] were used to match the HaldenPSFs and rating scale to the SPAR-H PSFs and rating scale. The dataare presented in Table 2. Each row represents a single crew'sperformance on one of the four HFEs. The first column of Table 2contains the HFE number for each data point.

The second column, “Error?” documents the outcome of thetrials (whether the crew met the criteria for occurrence of ahuman failure even, according to the criteria in Section 5.2.1). Inthis column, “Yes” denotes than an HFE occurred,“No” denotes thatthere was not an HFE (i.e., that the crew successfully responded tothe scenario).4

The next eight columns of Table 2 are the levels of the SPAR-HPSFs, which fully specify the context for each experimental trial. Asstated earlier, there are 19,440 possible contexts in the SPAR-Hmethod. For brevity in this manuscript, we have assigned a letter(A–D) to each context represented in the experimental data toavoid repeating the full context in subsequent tables and discus-sion (e.g., context “A” is used to refer [Time ¼ Extra, Stress ¼Nominal, Complexity ¼ Moderate,…, Work Processes ¼ Nominal],and so on).

At this point, it is important to differentiate between scenariosand contexts. The scenarios denote the experimental conditions –

in this case, there were four experimental scenarios (denoted 1A,1C, 2, 3). The contexts are the levels of the PSFs associated witheach crew in each scenario. The context for each scenario is notnecessarily identical for each crew; while some of the PSFs (e.g.,procedures) will not vary within a scenario, several of the PSFs (e.g., experience/training) could vary across crews. The data set inTable 2 includes four contexts. For this set of data, the contexthappens to be identical across all of crews within each scenario.This reflects the similarity of the four crews that participated inthe experiments. However, this uniformity may not be seen infuture simulator experiments with different crews.

5.4. Bayesian updating

The Bayesian updating approach described in Section 4 isimplemented on each of the contexts in SPAR-H. For the eightPSFs in SPAR-H, there are 19,440 possible contexts. For brevity, weonly demonstrate the Bayesian inference process for the fourcontexts represented in the extracted data.

The SPAR-H equations, Eqs. (7) and (8), deterministically assign anHEP for each combination of PSFs, PðErrorjPSF1 \ PSF2 \ ⋯ \ PSF8Þ.Relating this back to Section 4.3, SPAR-H assigns a value for Eðp0Þ foreach of 19,440 the possible contexts. To obtain Eðp0Þ for each context,we developed a Matlab script implementing Eqs. (7) and (8), the rulefor selecting the appropriate equation, and the lower limit on HEP.

We used a second script to assign the parameters of a CNI priordistribution for each p0. The parameters of the prior distributionfor the four contexts represented in the data are shown in Table 3.

To implement the Bayesian approach we need two pieces ofdata for each context: x, the number of human failures, and n, thenumber of opportunities for human failure. The data for each

Table 2SPAR-H PSF assignments and performance outcomes extracted from the 15 Halden simulator experiments. See Section 5.3 for description of each column.

HFE Error? Time Stressors Complex. Exper. Procedures HMI Fitness WorkProc Context

1A No Extra Nominal Moderate Nominal Avail., but poor Nominal Nominal Nominal A1A No Extra Nominal Moderate Nominal Avail., but poor Nominal Nominal Nominal A1A No Extra Nominal Moderate Nominal Avail., but poor Nominal Nominal Nominal A1A No Extra Nominal Moderate Nominal Avail., but poor Nominal Nominal Nominal A1C No (No) Barely adeq. High Moderate Nominal Avail., but poor Nominal Nominal Nominal B1C No (Yes) Barely adeq. High Moderate Nominal Avail., but poor Nominal Nominal Nominal B1C No (Yes) Barely adeq. High Moderate Nominal Avail., but poor Nominal Nominal Nominal B1C Yes (Yes) Barely adeq. High Moderate Nominal Avail., but poor Nominal Nominal Nominal B2 Yes Inadequate High High Low Avail., but poor Nominal Nominal Poor C2 Yes Inadequate High High Low Avail., but poor Nominal Nominal Poor C2 Yes Inadequate High High Low Avail., but poor Nominal Nominal Poor C2 Yes Inadequate High High Low Avail., but poor Nominal Nominal Poor C3 No Extra Nominal Nominal Nominal Nominal Nominal Nominal Nominal D3 No Extra Nominal Nominal Nominal Nominal Nominal Nominal Nominal D3 No Extra Nominal Nominal Nominal Nominal Nominal Nominal Nominal D

4 The error column for HFE 1C contains two different outcomes for each crew,reflecting two different failure criteria. For HFE 1C, the plant failure criterion is“failure to isolate the ruptured SG and control pressure below the SG PORV setpointto avoid SG PORV opening”; this HFE interpretation is used in many PRAs. In theexperiments, the failure criterion added a 45 min time cut-off to the PRA criterion;the 40 min cut-off is not a system requirement. In the experiment, some crews metthe system needs, but did not do it within 40 min, leading to expert disagreementabout whether the outcome was an HFE. In the error column in Table 2, the firstresult corresponds to the PRA failure criterion and parenthetical result correspondsto the experimental failure criterion. This problem demonstrates one of the known

(footnote continued)difficulties facing HRA: inconsistent HFE definitions. While Bayesian methodscannot resolve the definition of an HFE, they can be used to combine this uncertaindata. In the current paper the analysis is conducted using both interpretations.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–40 37

context in Table 2 can be aggregated together to provide values forx and n. The aggregated data for the four contexts are presented inTable 4. Two sets of data are presented for context B, to account forthe two sets of failure criteria for Halden scenario 1C. The firstset of data is for human errors, as defined by the PRA criteria.The second set of data is for human errors, as defined by theexperimental criteria.

To obtain the posterior distribution, we apply Eq. (5) tocombine the prior and the data. The posterior distribution for p1for the four updated contexts is presented in Table 4.

5.5. Results

Table 4 demonstrates that it is possible to use sparse data torefine HEPs for existing HRA methods. Furthermore, comparingthe posteriors (Table 4) with the priors (Table 3) shows theimportance of incorporating data into existing HRA methods.

For contexts A, C, and D, the posterior mean is very close to theprior mean. For contexts A and D, we did not see any errors –

0 failures in 4 trials and 0 failures in 3 trials, respectively. Incontext A, the mean HEP shifted only slightly (from 1.00E�3 to9.92E�4), and in context D the mean did not change (it remains1.00E�4). For both contexts the prior distribution is relativelystrong, and the amount of data is small. While four data points willnot significantly shift a strong prior, in both cases the newevidence suggest that the failure probability should be low, whichprovides enhanced technical basis for these HEP assignments.Similarly, in context C, which has a strong prior, adding evidenceof 4 failures in 4 trials resulted in no change in the mean HEP of1.0. However, the new evidence supports assignment of highfailure probabilities for context C. For these three contexts,combining data with the SPAR-H method does not change theHEPs assigned. However, adding the data enhances the technicalbasis of the SPAR-H HEPs, and this enhanced technical basisincreases the credibility of the SPAR-H method.

For context B, the prior distribution is more diffuse than thepriors for contexts A, C, and D, which indicates higher uncertainty;for this prior, four trials could change the distribution. For both

interpretations of the data for context B, the posterior mean ismeasurably different than the prior mean, and in both cases, theHEPs increase. For case BPRA, the posterior mean is a slight increaseover the prior mean. For case Bexp, the posterior mean increasesmore substantially. In the BPRA case, adding new data points alsoresulted in a more peaked probability distribution, which reducesuncertainty about the HEP. While the mean HEP for increased, the95th percentile decreased from 0.57 to 0.49.

While it is essential to conduct more rigorous evaluation of thecompatability of Halden data and SPAR-H before making definitivestatements about the performance of SPAR-H, this example illus-trates how Bayesian methods and simulator data can be used toimprove the qualitative and quantitative basis of HRA. In allcontexts, adding data increases the credibility of the estimates inexpert-derived HRA methods. As demonstrated by context B, evensmall amounts of new data can make a quantitative impact on theHEP estimates in existing HRA methods. Exercises such as thisexample can illuminate areas where HEP assignments are consis-tent with data, and areas where the HEPs assigned by a methodmay be non-conservative. Using Bayesian updating to add data toHRA methods like SPAR-H also bring the expert-informed HEPsmore in line with the data that are observed.

6. Discussion

Adding the data enhances the technical basis of the SPAR-HHEPs, and this enhanced technical basis increases the credibility ofthe SPAR-H method.

The results of the case study demonstrate that Bayesianupdating can be used to refine the probability assignments fromexisting HRA methods. In this case, adding simulator data to theSPAR-H method enhanced support for several of the HEP assign-ments, and also illustrated a potential non-conservative HEP.While the case study needs further evaluation before makingofficial changes to the SPAR-H methodology, the potential non-conservative HEP assignment provides a powerful argument forthe need to use data to refine existing HRA methods.

6.1. Applicability of simulator data

Simulator data provide powerful insight into PðErrorjPSFsÞ.Leveraging simulators provides a unique opportunity to gather,via controlled experiments, data for events that would notnormally be seen in the historical record for NPP operation (e.g.,severe accidents, or unlikely PSF states such as unavailableprocedures). These controlled experiments can be designed tofocus on specific causal factors in order to make inference onperformance under known conditions.

It can be argued that simulator data do not always representthe expected conditions in a nuclear power plant (e.g., thesimulator events are designed to be more difficult than “real”accidents). However, the approach presented in this paperbypasses this difference, by reasoning about PðErrorjPSFsÞ ratherthan P(Error). While the simulator data are not necessarily per-fectly representative of the context of real operational events, thesimulator data are fully representative of the occurrence of humanfailure events, given the PSFs. That is, we believe that operatorsresponding to a given context will exhibit the same response inboth real events and in simulator performance – and this assump-tion forms the basis of the simulator-based training programs atNPPs around the world.

Bayesian methods also open the door to reasoning about P(Error), under known or unknown conditions. The use of BayesianNetworks permits users to reason about both P(Error) andPðPSFsjErrorÞ. The results of the current work could be

Table 3Expected value and prior distribution for p for each of the four contexts representedin the simulator data. The expected value is the HEP calculated from the SPAR-Hmethod. The expected value was used to fit the constrained non-informative priordistribution in the table.

Context Eðp0Þ Prior

A 1.00E�3 p0 � Betað0:5;499:5ÞB 1.67E�1 p0 � Betað0:5;2:4975ÞC 1.00 p0 � Betað50000;0:5ÞD 1.00E�4 p0 � Betað0:5;4999:5Þ

Table 4Aggregated data and posterior distribution for p for the four contexts representedin the simulator data. The relevant data are the number of failures, x, and thenumber of opportunities for failure, n. For context B, we ran the analysis for bothinterpretations of the data. The expected value of p1 is calculated from the posteriordistribution.

Context Data (x/n) Posterior Eðp1Þ

A 0/4 p1 � Betað0:5;503:5Þ 9.92E�4BPRA 1/4 p1 � Betað1:5;5:4975Þ 2.14E�1Bexp 3/4 p1 � Betað3:5;3:4975Þ 5.00E�1C 4/4 p1 � Betað50004;0:5Þ E1.00D 0/3 p1 � Betað0:5;5002:5Þ 1.00E�4

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–4038

implemented on the SPAR-H Bayesian Network model proposedby [27], resulting in an HRA framework which leverages Bayesianmethods and HRA data at multiple levels of abstraction.

6.2. Challenges: data access and organization

The greatest challenge experienced was gaining access to thedata. The most relevant information comes from a wide range ofsimulator data (several sources of such data are described inSection 4.2) and operational experience (including HERA [47],OPERA [15], and CORE-DATA [48]). These databases provide apowerful opportunity to update existing HRA methods. However,access to the data is severely limited due to the sensitivity of theraw data.

The effort taken by the Halden reactor project to anonymize theraw data resulted in an extremely rich set of information. To trulyenable use of data for HRA, future data collection projects mustfollow Halden's lead by anonymizing the data and making itavailable to research organizations.

The simulator experiments provide a good starting point fordata collection. It is imperative to continue obtaining simulatordata. However, the scope of HRA data must be expanded to includespecific human-performance elements (PSFs) in the operationaldata. Since HRA models have a variety of PSFs, the data required toupdate existing HRA methods must span the spectrum of PSFsused in current HRA methods. The PSF taxonomy proposed by [49]provides a comprehensive, method-independent starting point forexpanding operational data collection frameworks to include acomprehensive set of PSFs.

On a related note, there is a broad range of information, beyondsimulator and operational data, that is relevant to human perfor-mance in nuclear power plants. Possible sources of informationthat may be relevant to HRA hypotheses include existing HRAmethods, expert judgment, and cognitive literature. These infor-mation sources provide a variety of both qualitative and quanti-tative information, including point and interval estimates of HEPs,linear models, correlations between various PIFs, the magnitude ofPSF influences, and relationships between variables. The data fromthese sources may be less sensitive than the simulator andoperational data. Effort should be undertaken to assemble thisinformation in a database accessible to the HRA research commu-nity. There have been efforts to develop databases to aggregatethis type of information [50,51], but the value of these databaseswill be overlooked until there is a concerted effort to populatethem with relevant information.

6.3. Next steps

The case study in this paper only included data about four ofthe contexts in SPAR-H. As a next step for enhancing HRA, it isdesirable to continue to gather simulator data, ideally from multi-ple simulators, to capture the other contexts in the SPAR-Hmethod and to provide useful information for updating otherHRA methods. In parallel, it is important to critically consider therelationship between simulator data and current HRA methods.

The methodology discussed in this manuscript can also beextended to handle additional issues that will arise with the use ofsimulator data. For example, in Section 5.4, we presented resultsfor two different interpretations of the Halden data (contexts BPRAand Bexp). In a more complex Bayesian analysis, we could directlyaddress the uncertainty in the data by assigning prior distributionsto the number of failures in the data and propagating thesedistribution through the analysis.

For this case study, we used a CNI prior distribution, which isknown to have relatively light tails. This property makes it difficultto update with sparse data for low probability events. In future

work, researchers should follow the same methodology usingother diffuse priors (including mixture priors) and compare resultsto the CNI prior.

7. Conclusions

This paper presented a Bayesian methodology to update HEPsfrom existing HRA methods with simulator data. We provided acase study, wherein we updated several values from SPAR-H withsimulator data from Halden experiments. This methodology andcase study demonstrate the value of Bayesian methods for HRA.Furthermore, we demonstrated that we do not require an inordi-nate amount of data, nor do we require perfect data to improveexisting HRA approaches. The same Bayesian approach can be usedto add operational data to HRA methods, if the operational datainclude descriptions of the PSFs that are relevant to the opera-tional performance.

Additionally, by using the Bayesian Network framework, it ispossible to combine data about PðErrorjPSFsÞ with data about P(PSFs) to provide an expanded technical basis for HRA. The BNapproach has the added benefit of an expanded scope of reason-ing, enabling inference about P(Error) and PðPSFsjErrorÞ. The BNversion of the SPAR-H model, documented in [27] can be updatedusing the same methodology documented in this paper. Using theBayesian updating on the probabilities produced by BN modelsprovides the ability to incorporate a wide variety of data sources toenhance the credibility of the model.

The Bayesian inference process can be applied to a wide rangeof HRA problems. In this paper, we demonstrated how to conductinference for HRA problems at a high level of abstraction (failureprobabilities for general human tasks). Future works should bedeveloped to demonstrate how the Bayesian inference process canbe used to conduct inference on HRA problems at more detailed,causal levels (e.g., what is the effect of a given PSF on performance).

Implementing the Bayesian methodology brings HRA in linewith the Bayesian processes used in other aspects of PRA. Byexpending effort to bridge the gap between hardware and humanfailure modeling, HRA quality (and thereby PRA quality) isenhanced. Inherent deficiencies in any part of PRA degrade theapplicability of the PRA results. The Bayesian approach provides away to incorporate data into the HRA process, which will go a longway to resolve the model credibility issues in HRA.

Acknowledgments

We thank Andreas Bye and Michael Hildebrandt of the HaldenReactor Project for providing the anonymized experimental dataand for reviewing this manuscript. The authors also thank SusanStevens-Adams of Sandia National Laboratories for assistance withdata extraction. This work was supported by the LaboratoryDirected Research and Development program at Sandia NationalLaboratories. Sandia National Laboratories is a multi-programlaboratory managed and operated by Sandia Corporation, a whollyowned subsidiary of Lockheed Martin Corporation, for the U.S.Department of Energy's National Nuclear Security Administrationunder contract DE-AC04-94AL85000.

References

[1] Siu NO, Kelly DL. Bayesian parameter estimation in probabilistic risk assess-ment. Reliab Eng Syst Saf 1998;62(1):89–116.

[2] Atwood CL, LaChance JL, Martz HF, Anderson DJ, Englehardt M, Whitehead D,et al. Handbook of parameter estimation for probabilistic risk assessment.NUREG/CR-6823; US Nuclear Regulatory Commission; Washington DC; 2003.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–40 39

[3] Kelly DL, Smith CL. Bayesian inference in probabilistic risk assessment—thecurrent state of the art. Reliab Eng Syst Saf 2009;94(2):628–43.

[4] Kelly D, Smith C. Bayesian inference for probabilistic risk assessment: apractitioner's guidebook. Springer, London; 2011.

[5] Podofillini L, Dang VN. A Bayesian approach to treat expert-elicited probabil-ities in human reliability analysis model construction. Reliab Eng Syst Saf2013;117:52–64.

[6] French S, Bedford T, Pollard SJT, Soane E. Human reliability analysis: a critiqueand review for managers. Saf Sci 2011;49(6):753–63.

[7] Hildebrandt M, Oxstrand J, Boring RL. Living HRA: building new communitiesof practice for proactive safety management. In: Proceedings of the humanfactors and ergonomics society 53rd annual meeting, vol. 53. Human Factorsand Ergonomics Society; 2009. p. 1579–83.

[8] Adhikari S, Bayley C, Bedford T, Busby JS, Cliffe A, Devgun G, et al. Humanreliability analysis: a review and critique. Manchester Business School Work-ing Paper No. 589; Manchester Business School, The University of Manchester;Manchester, UK; 2009. URL ⟨http://www.mbs.ac.uk/research/workingpapers/⟩.

[9] Committee on the Safety of Nuclear Installations (CSNI) Working Group onRisk (WGRisk). HRA data and recommended actions to support the collectionand exchange of HRA data, NEA/CSNI/R(2008)9; OECD Nuclear Energy Agency;2008.

[10] Mosleh A, Chang YH. Model-based human reliability analysis: prospects andrequirements. Reliab Eng Syst Saf 2004;83(2):241–53.

[11] Dougherty EM. Human reliability analysis—where shouldst thou turn? ReliabEng Syst Saf 1990;29(3):283–99.

[12] Committee on the Safety of Nuclear Installations (CSNI). Simulator studies forHRA purposes, NEA/CSNI/R(2012)1; OECD Nuclear Energy Agency; 2012,Budapest, Hungary.

[13] Skjerve AB, Bye A, editors. Simulator-based human factors studies across 25years: the history of the Halden man-machine laboratory. Springer, London;2010.

[14] Broberg H, Hildebrandt M, Nowell R. Results from the 2010 HRA datacollection at a US PWR training simulator. HWR-981; OECD Halden ReactorProject; Halden, Norway; 2011.

[15] Park J, Jung W. OPERA—a human performance database under simulatedemergencies of nuclear power plants. Reliab Eng Syst Saf 2007;92(4):503–19.

[16] Chang YHJ, Lois E, Overview of the NRC's HRA data program and recentactivities. In: Proceedings of the international conference on probabilisticsafety assessment and management (PSAM 11), Helsinki, Finland; 2012.

[17] Chang YHJ, Mosleh A, Roth EM, Richards R, Shen SH, Bley D, et al. Methodologyfor collection and analysis of simulator data for HRA applications. In:Proceedings of the international conference on probabilistic safety assessmentand management (PSAM 11), Helsinki, Finland; 2012.

[18] Hallbert B, Kolaczkowski A. The employment of empirical data and Bayesianmethods in human reliability analysis: a feasibility study, NUREG/CR-6949; USNuclear Regulatory Commission; Washington DC; 2007.

[19] Kim MC, Seong PH, Hollnagel E. A probabilistic approach for determining thecontrol mode in CREAM. Reliab Eng Syst Saf 2006;91(2):191–9.

[20] Trucco P, Cagno E, Ruggeri F, Grande O. A Bayesian belief network modellingof organisational factors in risk analysis: a case study in maritime transporta-tion. Reliab Eng Syst Saf 2008;93(6):845–56.

[21] Groth KM. A data-informed model of performance shaping factors for use inhuman reliability analysis [Ph.D. thesis], University of Maryland, College Park,MD; 2009. ⟨http://gradworks.umi.com/33/91/3391233.html⟩.

[22] Baraldi P, Conti M, Librizzi M, Zio E, Podofillini L, Dang VN. A Bayesian networkmodel for dependence assessment in human reliability analysis. In: Proceed-ings of the European society for reliability annual meeting (ESREL 2009),Prague; 2009.

[23] Wang YF, Xie M, Ng KM, Habibullah MS. Probability analysis of offshore fire byincorporating human and organizational factor. Ocean Eng 2011;38:2042–55.

[24] Groth KM, Mosleh A. Deriving causal Bayesian networks from humanreliability analysis data: a methodology and example model. J Risk Reliab2012;226(4):361–79.

[25] Mindock J. Development and application of spaceflight performance sharpingfactors for human reliability analysis [Ph.D. thesis], University of Colorado,Boulder, CO; 2012.

[26] Vinnem JE, Bye R, Gran BA, Kongsvik T, Nyheim OM, Okstad EH, et al. Riskmodelling of maintenance work on major process equipment on offshorepetroleum installations. J Loss Prev Process Ind 2012;25(2):274–92.

[27] Groth KM, Swiler LP. Bridging the gap between HRA research and HRApractice: a Bayesian network version of SPAR-H. Reliab Eng Syst Saf2013;115:33–42.

[28] Sundaramurthi R, Smidts C. Human reliability modeling for the next genera-tion system code. Ann Nucl Energy 2013;52:137–56.

[29] Martins MR, Maturana MC. Application of Bayesian belief networks to thehuman reliability analysis of an oil tanker operation focusing on collisionaccidents. Reliab Eng Syst Saf 2013;110:89–109.

[30] Gertman D, Blackman H, Marble J, Byers J, Smith C. The SPAR-H humanreliability analysis method, NUREG/CR-6883; US Nuclear Regulatory Commis-sion; Washington DC; 2005.

[31] Swain AD. Accident sequence evaluation program human reliability analysisprocedure, NUREG/CR-4772; US Nuclear Regulatory Commission; 1987,Washington, DC.

[32] Parry GW, Beare AN, Spurgin AJ, Moieni P. An approach to the analysis ofoperator actions in probabilistic risk assessment. EPRI TR-100259; The ElectricPower Research Institute (EPRI); 1992, Palo Alto, CA.

[33] Reer B, Dang VN. The commission errors search and assessment (CESA)method. PSI Report Nr. 07-03; Paul Scherrer Institut; Switzerland; 2007.

[34] Hollnagel E. Cognitive Reliability and error analysis method (CREAM). NewYork: Elsevier; 1998.

[35] Williams JC. HEART—a proposed method for assessing and reducing humanerror. In: 9th Advances in reliability technology symposium. University ofBradford; 1986.

[36] Jung W, et al. A standard HRA method for PSA in nuclear power plant; K-HRAmethod, KAERI/TR-2961/2005; Korean Atomic Energy Research Institute(KAERI); 2005, Daejeon, Korea.

[37] Embrey DE, Humphreys P, Rosa EA, Kirwan B, Rea K. SLIM-MAUD: an approachto assessing human error probabilities using structured expert judgement,NUREG/CR-3518; US Nuclear Regulatory Commission; 1984, Washington, DC.

[38] Swain AD, Guttmann HE. Handbook of human reliability analysis withemphasis on nuclear power plant applications, NUREG/CR-1278; US NuclearRegulatory Commission; Washington DC; 1983.

[39] Cooper se, Ramey-Smith AM, Wreathall J, Parry GW, Bley DC, Luckas WJ, et al.A technique for human error analysis ATHEANA, NUREG/CR-6350; US NuclearRegulatory Commission; Washington DC; 1996.

[40] Park J, Jung W. A database for human performance under simulated emer-gencies of nuclear power plants. Nucl Eng Technol 2005;37(5):491.

[41] Spurgin AJ, Moieni P, Gaddy CD, Parry G, Orvis DD, Spurgin JP, et al. Operatorreliability experiments using power plant simulators. EPRI NP-6937, vol. 1.Monterey, CA: Electric Power Research Institute; 1990.

[42] Prvakova S, Dang VN. A review of the current status of HRA data. In:Proceedings of the European society for reliability annual meeting (ESREL2013). Amsterdam; 2013.

[43] Atwood CL. Constrained noninformative priors in risk assessment. Reliab EngSyst Saf 1996;53(1):37–46.

[44] Lois E, Dang VN, Forester J, Broberg H, Massaiu S, Hildebrandt M, et al.International HRA empirical study – phase 1 report: description of overallapproach and pilot phase results from comparing HRA methods to simulatorperformance data, NUREG/IA-0216; Washington DC: US Nuclear RegulatoryCommission; 2009.

[45] Bye A, Dang VN, Forester J, Hildebrandt M, Marble J, Liao H, et al. Overviewand preliminary results of the U.S. empirical HRA study. In: Proceedings of theinternational conference on probabilistic safety assessment and management(PSAM 11), Helsinki, Finland; 2012.

[46] Form ES-604-2: Simulator crew evaluation form, NUREG-1021 rev. 9; USNuclear Regulatory Commission; 2007.

[47] Hallbert B, Boring R, Gertman D, Dudenhoeffer D, Whaley A, Marble J, et al.Human events repository analysis (HERA) system overview. NUREG/CR-6903,vol. 1. Washington DC: US Nuclear Regulatory Commission; 2006.

[48] Kirwan B, Basra G, Taylor-Adams S. CORE-DATA: a computerised human errordatabase for human reliability support. In: Proceedings of the 1997 IEEE sixthconference on human factors and power plants. Orlando, FL; 1997. p. 9.7–9.12.

[49] Groth KM, Mosleh A. A data-informed PIF hierarchy for model-based humanreliability analysis. Reliab Eng Syst Saf 2012;108:154–74.

[50] Boring RL, Gertman DI, Whaley AM, Joe JC, Tran TQ, Keller MD, et al.A taxonomy and database for capturing human reliability and humanperformance data. In: Proceedings of the human factors and ergonomicssociety annual meeting, vol. 50. Sage Publications; 2006. p. 2217–21.

[51] Park J, Dang VN. Performance factors for the analysis of crew responses tonuclear power plant simulated emergencies. In: Proceedings of the Europeansociety for reliability annual meeting (ESREL 2013). Amsterdam; 2013.

K.M. Groth et al. / Reliability Engineering and System Safety 128 (2014) 32–4040