Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1319https://doi.org/10.26868/25222708.2017.336

A Comparison of MCMC Algorithms for the

Bayesian Calibration of Building Energy Models

for Building Simulation 2017 Conference

Adrian Chong1, Khee Poh Lam1

1Center for Building Performance and Diagnostics, Carnegie Mellon University, Pittsburgh, USA

Abstract

Random walk Metropolis and Gibbs sampling areMarkov Chain Monte Carlo (MCMC) algorithms thatare typically used for the Bayesian calibration ofbuilding energy models. However, these algorithmscan be challenging to tune and achieve convergencewhen there is a large number of parameters. An alter-native sampling method is Hamiltonian Monte Carlo(HMC) whose properties allow it to avoid the ran-dom walk behavior and converge to the target distri-bution more easily in complicated high-dimensionalproblems. Using a case study, we evaluate the ef-fectiveness of three MCMC algorithms: (1) randomwalk Metropolis, (2) Gibbs sampling and (3) No-U-Turn Sampler (NUTS) (Hoffman and Gelman, 2014),an extension of HMC. The evaluation was carried outusing a Bayesian approach that follows Kennedy andO’Hagan (2001). We combine field and simulationdata using the statistical formulation developed byHigdon et al. (2004). It was found that NUTS is moreeffective for the Bayesian calibration of building en-ergy models as compared to random walk Metropolisand Gibbs sampling.

Introduction

Detailed building energy models have been increas-ingly used in the analysis of building energy con-sumption and the evaluation of energy conservationmeasures. To ensure its reliability, model calibrationhas been recognized as an integral component to theoverall analysis. A detailed description of the build-ing’s geometry, it’s associated HVAC system, and thequantification of various internal loads is typically re-quired as inputs to the model. However, detailedinformation is seldom available. Inarguably, uncer-tainty quantification becomes an important processin the use of detailed building energy models. Conse-quently, issues such as the calibration of input param-eters, prediction accuracy, and prediction uncertaintywould be of particular interest.

Many approaches for calibrating building energymodels have been proposed, requiring various degreesof automation, manual tuning and expert judgment(Coakley et al., 2014). In particular, there has been

increasing efforts in a Bayesian approach for the cal-ibration of building energy models (Heo et al., 2012,2015; Manfren et al., 2013; Chong and Lam, 2015; Liet al., 2016). This is because of its ability to quan-tify uncertainties in input parameters while at thesame time reducing discrepancies between simulationoutput and physical measurements. In the Bayesiancalibration of building energy models, Markov ChainMonte Carlo (MCMC) methods are a common wayfor sampling from the posterior distributions of thecalibration parameters. Its widespread use can beattributed to its ease of use in a wide variety ofproblems. Two basic MCMC algorithms are randomwalk Metropolis and Gibbs sampling. random walkMetropolis is routinely used in the Bayesian calibra-tion process due to its simple implementation. Therandom walk Metropolis algorithm (Metropolis et al.,1953) can be summarized as follows:

1. Arbitrarily select a valid initial starting point t0.

2. Suppose t0, t1, ..., ti have been generated. Gener-ate a candidate value t∗ from a symmetric pro-posal distribution given ti.

3. Calculate the Metropolis acceptance probabilityr, the probability of transitioning to the new can-didate value

r = min

{p(t∗|y)

p(ti|y), 1

}(1)

4. Accept and set ti+1 to the new candidate valuewith probability r or stay at the same point withprobability 1− r.

ti+1 =

{t∗ with probability r

ti with probability 1− r(2)

Gibbs sampling (Geman and Geman, 1984) proceedsby sampling each parameter from its conditional dis-tribution while holding the remaining parametersfixed at their current values. To illustrate, supposethere are d parameters t1, t2, ..., td. At each iterationi, Gibbs sampling cycles through each parameter tj ,and samples it from its conditional distribution given

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1320

the current value of the other parameters. This canbe expressed by the following equation:

tij ∼ p(tj |ti1, ..., tij−1, ti−1j+1, ..., ti−1d ) (3)

where ti1, ..., tij−1, t

i−1j+1, ..., t

i−1d represents all other pa-

rameters at their current values except tj .An alternative sampling method that has been gain-ing interest is Hamiltonian Monte Carlo (HMC).HMC avoids the random walk behavior inherent inrandom walk Metropolis algorithm and Gibbs sam-pling by using first-order gradient information to de-termine how it moves through the target distribu-tion (Hoffman and Gelman, 2014). The properties ofHMC allows it to converge to the target distributionmore quickly in complicated high-dimensional prob-lems (Neal, 1993). However, HMC requires users toprovide values of two hyperparameters: a step size εand the number of steps L, making it difficult andtime consuming to tune. To mitigate the challengesof tuning, the No-U-Turn Sampler (NUTS) was de-veloped by Hoffman and Gelman (2014). NUTS usesa recursive algorithm to automatically tune the HMCalgorithm without requiring user intervention or timeconsuming tuning runs was used.Previous studies have been focused on the applica-tion of Bayesian calibration to building energy mod-els without sufficient emphasis on the inference andassessment of convergence. Currently, the Bayesiancalibration of a building energy model is considered tobe complete when the model’s output meets the errorcriteria set out by ASHRAE guideline 14 (ASHRAE,2002). However, if the MCMC algorithm has notproceeded long enough, the generated samples maybe grossly unrepresentative of the target posteriordistributions (Gelman et al., 2014). In this paper,the objective is to evaluate the effectiveness of threeMCMC algorithms (random walk Metropolis, Gibbssampling and NUTS) within the Bayesian calibrationframework by Kennedy and O’Hagan (2001).

Method

We evaluate the effectiveness of three MCMC algo-rithms (random walk Metropolis, Gibbs sampling andNUTS) by applying each algorithm to a Bayesian cal-ibration approach that follows Kennedy and O’Hagan(2001). The process is as follows:

1. Build EnergyPlus model using construction draw-ings, design specifications and measured data

2. Conduct sensitivity analysis to reduce number ofcalibration parameters and avoid overfitting themodel. Train a Gaussian process (GP) emulatorto map the energy model’s input parameters tothe model output of interest.

3. Apply Bayesian calibration to the GP emulator(Kennedy and O’Hagan, 2001). The Bayesian cal-ibration process would be repeated using differentMCMC algorithms.

LoadProfile:Plant

Chilled Water Pump

Chiller 1

Chiller2

Cooling Tower 1

Cooling Tower 2

Condenser Water Pump

Chilled Water Loop

Condenser Loop

Figure 1: Diagram of cooling system modelled in En-ergyPlus.

4. Compare the effectiveness of different MCMCalgorithm using trace plots and Gelman-Rubinstatistics to diagnose convergence to the posteriordistribution.

We illustrate each step in the subsequent subsectionswith a case study.

EnergyPlus modelAs a first step, the cooling system of a large ten-story office building located in Pennsylvania U.S.Awas modelled using EnergyPlus version 8.5. The En-ergyPlus model was built based on construction draw-ings, design specifications and site visits, and consistsof the following functional parts (Figure 1): (a) Loadsfrom cooling coil that transfers heat from air to wa-ter; (b) Two chillers connected in parallel that coolsthe water; (c) Chilled-water distribution pumps thatsend chilled water to the loads; (d) Condenser wa-ter pumps for circulation in the condenser loop; and(e) Two cooling towers in parallel that rejects heatfrom the chillers to the atmosphere. The Energy-Plus objects used to model these components includethe LoadProfile:Plant object, the Chiller:Electric:EIRobject, the Pump:VariableSpeed object and the Cool-ingTower:SingleSpeed object. Initial values were as-signed to the model parameters based on measureddata and design specifications (Table 1).The LoadProfile:Plant object is used to simulate ascheduled demand profile when the coil loads are al-ready known (LBNL, 2016b). This makes it possibleto isolate and calibrate the HVAC system withoutany propagation of uncertainties due to calculation

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1321

of building loads. Hourly demanded loads were calcu-lated based on the following equation (LBNL, 2016a):

Qload = mcp(Tin − Tout) (4)

where Tout and Tin denotes the outlet and inlet watertemperature respectively; Qload is the scheduled coilload; m is the mass flow rate; and cp is the specificheat of water .The Chiller:Electric:EIR object uses performance in-formation at reference conditions along with threeperformance curves to determine the chiller’s per-formance at off-reference conditions (LBNL, 2016a).The three performance curves are: (1) CoolingCapacity Function of Temperature Curve (CapFT)(Equation 5), (2) Energy Input to Cooling Out-put Ratio Function of Temperature Curve (EIRFT)(Equation 6), and (3) Energy Input to Cooling Out-put Ratio Function of Part Load Ratio Curve (EIRF-PLR) (Equation 7).

CapFT = a1 + b1(Tcw,l) + c1(Tcw,l)2+

d1(Tcond,e) + e1(Tcond,e)2 + f1(Tcw,l)(Tcond,e)

(5)

EIRFT = a2 + b2(Tcw,l) + c2(Tcw,l)2+

d2(Tcond,e) + e2(Tcond,e)2 + f2(Tcw,l)(Tcond,e)

(6)

EIRFPLR = a3 + b3(PLR) + c3(PLR)2 (7)

Tcw,l and Tcond,e denotes the leaving chilled watertemperature and entering condenser fluid tempera-ture respectively; Qref and COPref are the chiller’scapacity and coefficient of performance (COP) at ref-erence conditions; and PLR is the chiller part-loadratio and equals cooling load

(Qref )(CAPFT ) . Using Equations 5

to 7, chiller power under a specific operating condi-tion can be determined by the following equation.

Pchiller =Qref

COPref(CapFT )(EIRFT ) (8)

Inputs to this chiller model (Qref , COPref , regres-sion coefficients of Equations 5, 6 and 7) were deter-mined based on measured data using the reference-curve method that was proposed by Hydeman andGillespie Jr (2002).The Pump:VariableSpeed object calculates the powerconsumption of a variable speed pump using a cubiccurve (Equation 9) (LBNL, 2016b).

FFLP = a5+b5(PLR)+c5(PLR)2+d5(PLR)3 (9)

where PLR = Flow RateDesign Flow Rate . Using Equation 9,

pump power is calculated by the following equation.

Ppump = (Pdesign)(FFLP )(Effmotor) (10)

where Pdesign is the design power consumption andEffmotor is the motor efficiency. Inputs to the pumpmodel were assigned using measurement of flow rateand pump power consumption. We assign a value of1 to motor efficiency because pump motor inefficien-cies are already accounted for in the measurementsof flow and power. We use least squares regressionto compute the coefficients of Equation 9 with PLRand FFLP calculated as follows:

FFLPi =poweri

max(power1, power2, ..., powern)(11)

PLRi =flowi

max(flow1, f low2, ..., f lown)(12)

Sensitivity analysis

Before calibrating the model, sensitivity analysis wasperformed to identify the parameters that have themost influence over the model’s output. The objec-tive is to reduce the number of calibration parametersand use only important factors in the calibration pro-cess. This not only reduces computation cost but alsohelps mitigate overfitting. Morris method (Morris,1991) was used to carry out the sensitivity analysis.This was executed with R sensitivity package (Pujolet al., 2016).Ten parameters in the cooling system were selectedas uncertain (Table 1). Although the set of uncer-tain parameters are specific to this case study, theycorrespond to the set of parameters typically selectedas random variables for the cooling system. All pa-rameters were assigned a uniform distribution. Pumpmotor efficiency was varied between 0.6 and 1.0. Theremaining 8 parameters were varied ±20% of theirinitial values. Design fan power and nominal capac-ity of cooling towers 1 and 2 were modeled as a singlerandom variable because they have the same makeand specification and were installed at the same time(Table 1). On the contrary, chillers 1 and 2 havevery different capacity and COP at reference condi-tions and hence were modeled as separate randomvariables. We use the modified mean µ∗ proposed byCampolongo et al. (2007) and standard deviation σ

Table 1: List of model parameters and their range.

Model parameter Symbol Initial Value Min MaxChiller 1:Reference Capacity θ1 653378 522702 784053Reference COP θ2 6.86 5.49 8.23

Chiller 2:Reference Capacity θ3 243988 195190 292785Reference COP θ4 2.32 1.85 2.78Chilled water pump:Design Power Consumption θ5 18190 14552 21828Motor Efficiency θ6 1.0 0.6 1.0Condenser water pump:Design Power Consumption θ7 11592 9274 13911Motor Efficiency θ8 1.0 0.6 1.0Cooling Tower 1 and 2:Design Fan Power θ9 11592 9274 13911Nominal Capacity θ10 549657 439726 659589

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1322

●

●●

●

●●●●●

●

0.0 0.1 0.2 0.3 0.4

0.00

0.02

0.04

0.06

0.08

µ*

σ

!₄

!₂ !₃

!₁

!₁₀

!₅ - !₉

Figure 2: Sensitivity analysis (Morris method) of pa-rameters in Table 1.

to determine which parameters are sensitive. Param-eters θ5−θ9 have µ∗ and σ of approximately zero indi-cating that they are negligible parameters and shouldbe excluded (Figure 2). Hence, only the top 5 param-eters (θ1, θ2, θ3, θ4, and θ10) would be used for theBayesian calibration of the EnergyPlus model.

Bayesian calibrationA Bayesian calibration approach that follows that ofKennedy and O’Hagan (2001) was employed for thisstudy. The formulation explicitly models uncertaintyin calibration parameters, uncertainty due to discrep-ancy between the simulator and actual physical sys-tem, and observation errors as follows:

y(x) = η(x, t) + δ(x) + ε(x) (13)

η(x, t) denotes the building energy simulator outputgiven input vector (x, t), where t represents the cali-bration parameters required as inputs to the energymodel computed at known conditions x. Note thatwe make a distinction between the uncertain param-eters θ and the calibration parameters t, where thecalibration parameters t refer to the parameters thatwere selected from the set of uncertain parametersθ based on the results of the sensitivity analysis asdescribed in the previous section. The term δ(x) isused to account for discrepancies between the sim-ulator η(x, t) and the actual physical system. ε(x)denotes observation error.Field data and simulation data were combined us-ing the statistical formulation developed by Higdonet al. (2004). Table 2 summarizes the data used toconstruct the field and simulation data for the casestudy. η(x, t) denotes the output of the EnergyPlussimulation which depends on the observable inputs tothe model x and the unknown calibration parameters

Table 2: Description of different parts used forBayesian calibration of the case study.

Symbol Descriptiony(x) Observed hourly cooling energy con-

sumption at corresponding values of xη(x, t) Hourly cooling energy consumption

prediction using EnergyPlus at corre-sponding values of observable inputs xand unknown calibration parameters t

x Observed hourly cooling coil load andchilled water flow rate

t Calibration parameters t1 = θ1, t2 =θ2, t3 = θ3, t4 = θ4 and t5 = θ10 (Table1 and Figure 2). Values were set usingLHS design

t. To learn about the calibration parameters t, Ener-gyPlus simulations were run at the same observableinputs x in our computer design of experiments. Thecorresponding calibration parameters t for the simu-lation runs were determined using Latin Hypercubesampling (LHS) to ensure sufficient coverage of theparameter space.Since the energy model is computationally expensiveto evaluate, a key element of this approach is theuse of a Gaussian Process (GP) model to carry outthe inference during the MCMC sampling procedure,mapping the energy model’s input parameters to themodel output of interest. A mean function µ(x, t) andcovariance function Cov((x, t), (x′, t′)) is required tospecify a GP model. For simplicity, we specify a meanfunction that is set to zero and a covariance functionthat follows Higdon et al. (2004) with the form:

Cov((x, t), (x′, t′)) =

1

ληexp

{−

p∑j=1

βηj |xij − x′ij |α −

q∑k=1

βηp+k|tik − t′ik|α

}(14)

where λη is the variance hyperparameter and βη is thecorrelation hyperparameter of the GP model. Thediscrepancy term δ(x) was also modelled as a GPmodel with mean function set to zero and a covari-ance function of the form:

Cov(x, x′) =1

λδexp

{−

p∑k=1

βδk|xik − x′ik|α}

(15)

α was set to 2 for both covaraiance functions. Fi-nally, observations errors ε(x) was modelled as Gaus-sian noise:

ε(x) ∼ N (0, I/λε) (16)

For estimating the calibration parameters (θ), cor-relation hyperparameters (βη and βδ), and variance

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1323

hyperparameters (λη, λδ and λε), we use MCMC toexplore and generate samples from their posterior dis-tributions.

Comparison of MCMC algorithms

We compare the effectiveness of three MCMC algo-rithms: NUTS (a variant of HMC) and the more com-monly used random walk Metropolis and Gibbs sam-pling. The comparison was carried out by checkingfor mixing and convergence to the target distributionusing the following metrics

• Trace plots: trace plots are plots of the chainsversus the sample index and can be useful forassessing convergence (Gelman et al., 2014). Ifthe distribution of points remains relatively con-stant, it suggests that the chain might have con-verged to the stationary distribution. A tracecan also tell you whether the chain is mixing well.

• Gelman-Rubin statistics (R): R is the ratio ofbetween-chain variance to within-chain varianceand is based on the concept that if multiplechains have converged, there should be little vari-ability between and within the chains (Gelmanet al., 2014). For convergence, R should be ap-proximately 1± 0.1.

For comparison, the same number of iterations wererun for all three algorithms. Four independentchains of 10,000 iterations per chain were run foreach MCMC algorithm with the first 5,000 iterations(50%) discarded as warmup/burn-in to reduce theinfluence of the starting values. For random walkMetropolis, an additional tuning of the acceptanceratio was required. It is generally accepted that theoptimal acceptance rate of the Metropolis algorithmis about 20% (Gelman et al., 1996). We used anormal proposal/jumping distribution and tuned itsvariance until an acceptance rate of between 20%and 25% was achieved. The random walk Metropolistook a shorter time to run than the NUTS for asingle chain of 10,000 iterations. However, afterconsidering the iterative tuning process, NUTS ranfaster since approximately three to four iterationswere required to achieve an acceptance rate of about20% with random walk Metropolis. Gibbs samplingtook significantly longer to run than NUTS andrandom walk Metropolis, since the algorithm cyclesthrough each parameter at each iteration.

Figures 3, 4, 5 and 6 provides a visual comparison ofthe trace plots (10,000 iterations including warmup)with samples generated by the three different MCMCalgorithms. Random walk Metropolis demonstratesbad mixing for the calibration parameters t (Figure3), indicating that the algorithm does not sufficientlyexplore the parameter space. After 10,000 iterationsthe calibration parameters have R between 1.89 and2.51 (Table 3), indicating that the variance betweenthe four independent chains are still greater than the

Table 3: R of calibration parameters with differentMCMC algorithms.

Parameters RandomMetropolis

GibbsSampling

NUTS(HMC)

t1 1.89 1.00 1.00t2 2.51 1.00 1.00t3 1.98 1.00 1.00t4 2.16 1.00 1.00t5 1.93 1.00 1.00

Table 4: R of hyperparameters with different MCMCalgorithms.

Hyper-parameters

RandomMetropolis

GibbsSampling

NUTS(HMC)

βη1 3.49 1.00 1.00βη2 2.73 1.00 1.00βη3 7.87 1.01 1.00βη4 1.57 1.01 1.00βη5 1.58 1.00 1.00βη6 2.94 1.01 1.00βη7 1.95 1.03 1.00βδ1 2.46 1.00 1.00βδ2 4.05 1.00 1.00λη 33.26 1.00 1.00λδ 302.38 1.05 1.00λε 1299.79 1.46 1.00

variance within. Gibbs sampling and NUTS performsbetter, with the calibration parameters t achievingadequate convergence (1±0.1) after 10,000 iterations.Trace plots also shows good mixing for both Gibbssampling and NUTS.

As expected, with random walk Metropolis, the cor-relation hyperparameters βη1−7 (Figure 4) and βδ1,2(Figure 5) of the GP model do not appear to be sta-ble. It is also clear that for βη1 , βη2 , βη3 , βδ1 and βδ2the different chains have not converged to a commondistribution. On the contrary, trace plots of sam-ples generated by Gibbs sampling and NUTS indi-cates rapid mixing for the correlation hyperpareme-ters βη1−7 (Figure 4) and βδ1,2 (Figure 5). Compar-

ing R for the correlation hyperparemeters, Table 4shows that after 10,000 iterations, the samples gener-ated by random walk Metropolis have not convergedyet (1.5 < R < 7.9). However, samples generatedby Gibbs sampling and NUTS have converged ade-quately with R within 1.0± 0.1 for all βη and βδ.

Figure 6 shows the trace plots for the variance hy-perparameters λη, λδ, and λε. From the figure, itcan be observed that with random walk Metropolis,the chains are moving very slowly (due to low accep-tance rates), and after 10,000 iterations the parallelsequences still have not converged to a common dis-tribution. Gibbs sampling performs better, showingrapid mixing for λη and slower but adequate mixingfor λδ. However, the trace plots show that λε is mov-ing very slowly through the parameter space, advanc-ing to the target distribution only after 5,000 itera-tions. The small step size also suggests poor mixingand that more iterations are needed to achieve ade-quate convergence. On the contrary, NUTS is able

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1324

Table 5: R of calibration parameters and hyperparam-eters with 50, 500 and 2000 iterations and 4 indepen-dent chains using NUTS and Gibbs sampling. Valuesexceeding 1± 0.1 are in red font

Number of Iterations50 500 2000

NUTS Gibbs NUTS Gibbs NUTS Gibbst1 1.00 1.05 1.00 1.03 1.00 1.07t2 1.10 1.06 1.00 1.01 1.00 1.01t3 1.06 1.20 1.01 1.01 1.00 1.01t4 1.00 1.49 1.00 1.04 1.00 1.01t5 1.03 1.04 1.00 1.01 1.00 1.00βη1 1.03 7.22 1.00 1.77 1.00 2.91βη2 1.01 1.84 1.00 1.20 1.00 1.43βη3 0.98 1.43 1.00 1.40 1.00 1.49βη4 0.98 1.30 1.00 1.03 1.00 1.02βη5 1.01 1.53 1.00 1.01 1.00 1.04βη6 1.02 1.57 1.00 1.05 1.00 1.05βη7 1.02 1.77 1.00 1.25 1.00 1.08βδ1 0.97 1.29 1.00 1.00 1.00 1.02βδ2 1.06 1.02 1.00 1.06 1.00 1.00λη 1.00 1.39 1.00 1.02 1.00 1.01λδ 1.11 17.97 1.00 2.15 1.00 1.42λε 0.98 179.00 1.00 38.66 1.00 11.08

generate samples of the variance hyperparameters ef-fectively, as shown by the rapid mixing within eachindependent chain. Additionally, after 10,000 itera-tions, samples generated by random walk Metropolisstill have very large R, suggesting that the algorithmneeds to be run much longer (Table 4). R for samplesgenerated by Gibbs sampling perform much better.However, R = 1.46 for λε suggests that slightly moreiterations is required before adequate convergence canbe achieved. Samples generated by NUTS performsthe best, having R = 1.00 for all the variance hyper-parameters.

To summarize, using random walk Metropolis resultsin very poor performance. After 10,000 iterations,none of the calibration parameters and hyperparam-eters achieved adequate convergence. R values werelarger than 1.1 and the trace plots also show poormixing. To achieve convergence with random walkMetropolis, the step size of the jumping distributionneeds to be further tuned. Bad mixing may also bedue to strong correlations in the parameter space.

Gibbs sampling show significantly better performancewith all calibration parameters and hyperparametersachieving adequate convergence, with the exceptionof λε. The trace plot and R of λε suggests that itis close to converging to the posterior distribution.Hence running Gibbs sampling for slightly more iter-ations should result in adequate convergence for allcalibration parameters and hyperparameters. NUTSshows the best performance with R values of ex-actly 1.00 for all calibration parameters and hyper-paramters, indicating adequate convergence. Traceplots of all samples generated by NUTS also showrapid mixing. Furthermore, NUTS require a lot less

iterations to converge (Table 5) due to the rapid mix-ing in the chains . After 50 iterations, R is alreadyclose to 1.00 for all calibration parameters and GPhyperparameters. With 500 iterations, the four inde-pendent chains have certainly achieved adequate con-vergence. In comparison to Gibbs sampling, after 50iterations, almost all parameters have not converged(Table 5). After 500 and 2000 iterations, the cali-bration parameters t have adequately converged butseveral GP hyperparameters still have R larger than1.1, indicating the variance between the four chainsare still larger than the variance within. This suggeststhat it is harder to achieve adequate convergence forthe GP hyperparameters and that users should paygreater attention to the assessing convergence of thesehyperparameters.

ConclusionThe effectiveness of three MCMC algorithms (randomwalk Metropolis, Gibbs sampling and NUTS) wereevaluated in this paper. An EnergyPlus model wasfirst built. This is followed by using sensitivity anal-ysis to reduce the number of calibration parameters.Measured data and simulation data were then com-bined using the statistical formulation developed byHigdon et al. (2004), which closely follows Kennedyand O’Hagan (2001) Bayesian calibration approach.Since the energy model is computationally expensiveto evaluate, a key element of this approach is the useof a Gaussian Process (GP) emulator. Each of thethree MCMC algorithms was separately used to es-timate the posterior distributions of the calibrationparameters (t) and the GP hyperparameters (βη, βδ,λη, λδ and λε).

From the trace plots and Gelman-Rubin statistics (R)of the samples generated by each algorithm, it wasfound that NUTS is able to achieve adequate conver-gence to the posterior distribution the fastest, withsignificantly lesser number of iterations. This studyshowed that using NUTS, it is possible to achieveadequate convergence with significantly lesser num-ber of iterations (approximately 500 iterations) andno hand tuning at all. Random walk Metropolisshowed the poorest performance with none of theparameters showing convergence after 10,000 itera-tions. Gibbs sampling showed significant improve-ments in sampling effectiveness as compared to ran-dom walk Metropolis but may require large numberof iterations to achieve convergence. In conclusion,this study showed that for the Bayesian calibration ofbuilding energy models, compared to the commonlyused random walk Metropolis and Gibbs sampling,NUTS was able to more effectively generate samplesfrom the posterior distributions of the calibration pa-rameters and GP hyperparameters.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1325

Figure 3: Trace plot of calibration parameters t (Table 1). Four independent chains of 10000 iterations perchain were run for each MCMC algorithm.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1326

Figure 4: Trace plots of correlation hyperparameters βη1 to βη7 . Four independent chains of 10000 iterations perchain were run for each MCMC algorithm.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1327

β!1

Random WalkMetropolis Gibbs Sampler NUTS (HMC)

β!2

Figure 5: Trace plots of correlation hyperparameters βδ1 and βδ2 . Four independent chains of 10000 iterationsper chain were run for each MCMC algorithm.

Figure 6: Trace plots of variance hyperparameters λη, λδ, and λε. Four independent chains of 10000 iterationsper chain were run for each MCMC algorithm.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

1328

ReferencesASHRAE (2002). Guideline 14-2002, measurement of

energy and demand savings. American Society ofHeating, Ventilating, and Air Conditioning Engi-neers, Atlanta, Georgia.

Campolongo, F., J. Cariboni, and A. Saltelli (2007).An effective screening design for sensitivity anal-ysis of large models. Environmental modelling &software 22 (10), 1509–1518.

Chong, A. and K. P. Lam (2015). Uncertainty anal-ysis and parameter estimation of hvac systems inbuilding energy models. In Proceedings of the 14thIBPSA Building Simulation Conference.

Coakley, D., P. Raftery, and M. Keane (2014). Areview of methods to match building energy sim-ulation models to measured data. Renewable andSustainable Energy Reviews 37, 123–141.

Gelman, A., J. B. Carlin, H. S. Stern, and D. B. Ru-bin (2014). Bayesian data analysis, Volume 2. Tay-lor & Francis.

Gelman, A., G. O. Roberts, W. R. Gilks, et al. (1996).Efficient metropolis jumping rules. Bayesian statis-tics 5 (599-608), 42.

Geman, S. and D. Geman (1984). Stochastic relax-ation, gibbs distributions, and the bayesian restora-tion of images. IEEE Transactions on pattern anal-ysis and machine intelligence 6, 721–741.

Heo, Y., G. Augenbroe, D. Graziano, R. T.Muehleisen, and L. Guzowski (2015). Scalablemethodology for large scale building energy im-provement: Relevance of calibration in model-based retrofit analysis. Building and Environ-ment 87, 342–350.

Heo, Y., R. Choudhary, and G. Augenbroe (2012).Calibration of building energy models for retrofitanalysis under uncertainty. Energy and Build-ings 47, 550–560.

Higdon, D., M. Kennedy, J. C. Cavendish, J. A.Cafeo, and R. D. Ryne (2004). Combining fielddata and computer simulations for calibration andprediction. SIAM Journal on Scientific Comput-ing 26 (2), 448–466.

Hoffman, M. D. and A. Gelman (2014). The no-u-turn sampler: adaptively setting path lengthsin hamiltonian monte carlo. Journal of MachineLearning Research 15 (1), 1593–1623.

Hydeman, M. and K. L. Gillespie Jr (2002). Tools andtechniques to calibrate electric chiller componentmodels/discussion. ASHRAE Transactions 108,733.

Kennedy, M. C. and A. O’Hagan (2001). Bayesiancalibration of computer models. Journal of theRoyal Statistical Society: Series B (StatisticalMethodology) 63 (3), 425–464.

LBNL (2016a). Energyplus engineering reference: thereference to energyplus calculations. US Depart-ment of Energy .

LBNL (2016b). Energyplus input output reference:the encyclopedic reference to energyplus input andoutput. US Department of Energy .

Li, Q., G. Augenbroe, and J. Brown (2016). Assess-ment of linear emulators in lightweight bayesiancalibration of dynamic building energy models forparameter estimation and performance prediction.Energy and Buildings 124, 194–202.

Manfren, M., N. Aste, and R. Moshksar (2013).Calibration and uncertainty analysis for computermodels–a meta-model based approach for inte-grated building energy simulation. Applied en-ergy 103, 627–641.

Metropolis, N., A. W. Rosenbluth, M. N. Rosenbluth,A. H. Teller, and E. Teller (1953). Equation ofstate calculations by fast computing machines. Thejournal of chemical physics 21 (6), 1087–1092.

Morris, M. D. (1991). Factorial sampling plans forpreliminary computational experiments. Techno-metrics 33 (2), 161–174.

Neal, R. M. (1993). Probabilistic inference usingmarkov chain monte carlo methods.

Pujol, G., B. Iooss, A. J. with contribu-tions from Khalid Boumhaout, S. D. Veiga,J. Fruth, L. Gilquin, J. Guillaume, L. Le Gratiet,P. Lemaitre, B. Ramos, T. Touati, and F. Weber(2016). sensitivity: Global Sensitivity Analysis ofModel Outputs. R package version 1.12.2.