Rodney F. WeiherNOAA Chief Economist

Tom TeisbergTeisberg Associates

04/22/23Program Planning & Integration

United States Department of CommerceNational Oceanic and Atmospheric Administration

Using Industrial Operating Models for Cost Benefit

Analysis 2007 MacArthur Foundation Benefit-Cost Analysis Conference

Using Industrial Operating Models for Cost Benefit Analysis 2 of 24

Organizations Like NOAA Must Evaluate Their

Programs• First, there must be a credible rationale for

funding a program from public revenue.• If a rationale exists, the program must also

have benefits that exceed its costs.• Cost-Benefit Analysis is used to determine

whether programs pass this test.

Using Industrial Operating Models for Cost Benefit Analysis 3 of 24

In NOAA, Weather Forecasting is an Important

Program• Benefits of weather forecasts can be

estimated in three general ways:• Stated Preference - forecast users are asked

to self-assess the benefits they receive • Data Analysis - benefits are inferred from

outcome data, with and without forecasts• Economic Modeling - benefits are estimated

by simulating decisions and outcomes

Using Industrial Operating Models for Cost Benefit Analysis 4 of 24

My Focus Here is on Economic Modeling

• To use this method, it is necessary to model decisions and outcomes that are often complicated and esoteric.

• Typically, an organization like NOAA lacks the resources and industry knowledge to do such modeling from scratch.

• Thus it becomes critical to find and adapt existing industrial operating models.

Using Industrial Operating Models for Cost Benefit Analysis 5 of 24

Example: Benefits of Forecasts Used by

Electricity Generators• Electricity load (use) is weather dependent,

especially in regions where air-conditioning is a large component of total load.

• Good temperature forecasts allow the most cost-effective set of plants to be used to serve the load.

• The benefits of temperature forecasts are the generation cost savings they create.

Using Industrial Operating Models for Cost Benefit Analysis 6 of 24

The Decision-Making here is Really Complicated

• A temperature forecast for the next 24 hours must be translated into an electricity load forecast for each specific electricity system.

• Then generating units must be selected to minimize cost, given the load forecast.

• Finally, real-time adjustments must be made to compensate for inevitable errors between forecasted and actual loads.

Using Industrial Operating Models for Cost Benefit Analysis 7 of 24

To Estimate Benefits, We Used Existing Operating

Models• A study by Hobbs et. al. [1] modeled the cost

savings from better load forecasts.• Hobbs used a model from Baldick [2] to

represent choice among generating units, given a load forecast.

• And Hobbs developed an electricity dispatch model to simulate real-time adjustments to load forecast errors.

Using Industrial Operating Models for Cost Benefit Analysis 8 of 24

We Still Needed a Link between Temperature and Load

Forecasts• In our study [3], we used an operating model

developed by one of our co-authors -- an electricity load forecaster known as Neural Electric Load Forecaster (NELF).

• NELF uses a variety of information, including the next day temperature forecast, to estimate next day electricity load.

• This provided the needed link between temperature forecasts and load forecasts.

Using Industrial Operating Models for Cost Benefit Analysis 9 of 24

We Picked Sites Representing Each of

Three US Regions• Since we needed both weather and load data

for specific utility service areas, there were limitations on the number and location of sites we could consider.

• We choose two sites each representing respectively the northern US, the southern US, and the western US.

Using Industrial Operating Models for Cost Benefit Analysis 10 of 24

At Each Site, We Used NELF to Model Four Weather

Forecasts• Naïve -- tomorrow’s weather will be the same

as today’s.• MAV -- Model Output Statistic Aviation, an

NWS guidance product.• NWS Forecast -- represents 7 to 11 percent

improvement relative to MAV• Perfect -- tomorrow’s actual weather is used

as the forecast for tomorrow.

Using Industrial Operating Models for Cost Benefit Analysis 11 of 24

Load Forecast Errors Decline as Weather Forecasts Improve

0

1

2

3

4

5

6

7

Naïve MAV NWS Perfect

Loa

d Fo

reca

st E

rror

(MA

PE)

SouthNorthWest

Using Industrial Operating Models for Cost Benefit Analysis 12 of 24

We Used Hobbs’ Study to Value Load Forecast

Improvements• Hobbs et. al. used actual and forecasted load

data from two electric utility systems, one in the northeast and one in the south.

• Hobbs considered two alternative configurations of generating plants to serve each type of load.

• This produced four cases spanning a range of real-world possibilities.

Using Industrial Operating Models for Cost Benefit Analysis 13 of 24

For Each Case, Hobbs Estimated Benefits of Better Load

Forecasts• Hobbs scaled load forecasting errors up or

down to simulate varying forecast quality.• Generation decisions were simulated using

Baldick’s plant commitment model coupled with an Hobbs’ electricity dispatch model.

• Generation costs were calculated and used to estimate percentage savings from better load forecasts.

Using Industrial Operating Models for Cost Benefit Analysis 14 of 24

We Averaged Hobbs’ Results Across Generating System

Types• We averaged Hobb’s results across his

alternative configurations of generating systems, for each of his two regions.

• This produced a southern system cost savings that we applied to our Southern region, and a northeastern system cost savings that we applied to our North and West regions.

Using Industrial Operating Models for Cost Benefit Analysis 15 of 24

We Next Needed US Electricity Generation Costs

by Region• Since Hobbs’ costs savings were expressed as

percentage cost savings, we needed to estimate regional generation costs.

• We allocated total US utility generation to our three regions (South, North, and West).

• For each region, we estimated costs using $20 per million watt-hours as a typical cost.

Using Industrial Operating Models for Cost Benefit Analysis 16 of 24

Electricity Benefits of Weather Forecasts Are

Mainly in South

0

20

40

60

80

100

120

140

Mill

ion

$ pe

r ye

ar

NWS vs Naïve Perfect vs NWS

SouthNorthWest

Using Industrial Operating Models for Cost Benefit Analysis 17 of 24

There are Reasons Why Benefits Are Mainly in

South• Weather forecasts reduce load forecast error

much more in the South (see earlier chart).• Hobbs results show much higher costs of load

forecast errors for his southern system.• Both these explanations in turn probably

reflect the fact that space cooling demand is weather sensitive and very important in the South.

Using Industrial Operating Models for Cost Benefit Analysis 18 of 24

Benefits of Improved Forecasts Can Be Estimated

Two Ways• The MAV and NWS forecasts differ by small

amounts, and so can be directly used to estimate benefits of small improvements from the current forecast quality.

• We can also fit a curve through the cases we estimated (Naïve, MAV, NWS, Perfect) and use this curve to interpolate benefits of a variety of alternative forecast qualities.

Using Industrial Operating Models for Cost Benefit Analysis 19 of 24

A One-Percent Improvement Has Similar

Benefits Both Ways • Estimating the benefit of a one-percent

improvement in forecast quality from the NWS and MAV cases implies a value of $1.3 million per year.

• Estimating this benefit from a fitted curve implies a value of $1.1 million per year.

Using Industrial Operating Models for Cost Benefit Analysis 20 of 24

We Can Also Estimate the Value of a One Degree

Improvement• Here we used the fitted curve, since a one

degree improvement represents a relatively large change in forecast quality (about 1/3 of existing error).

• Using the fitted curve, a one-degree improvement in forecast quality has a value of about $35 million per year.

Using Industrial Operating Models for Cost Benefit Analysis 21 of 24

These Benefits Can Be Expressed in Present Value

• Compare a one-time investment to improve forecasts by one-degree with its cost and the present value of it benefits.

• At 5% discount rate, the present value of a one-degree improvement in weather forecasts is about $700 million; one percent is $22-26 million.

Using Industrial Operating Models for Cost Benefit Analysis 22 of 24

Summary

• Overall, official NWS forecasts appear to produce total benefits for electricity generators of about $156 million per year, mostly in the south. A perfect forecast would add $70 million.

• These are benefits from plant scheduling, and do not include other possible benefits, e.g. plant maintenance decision-making.

Using Industrial Operating Models for Cost Benefit Analysis 23 of 24

Summary (Concluded)

• Benefits of a small improvement in weather forecast quality were estimated to be $1.1 to 1.3 million per year, per percentage point of error reduced.

• Benefits of a larger improvement (one degree, or around 1/3 of the current error) were estimated at about $35 million per year or about $700 million in present value.

Using Industrial Operating Models for Cost Benefit Analysis 24 of 24

References• [1] Hobbs, Benjamin F., et. al. Analysis of the Value for Unit

Commitment Decisions of Improved Load Forecasts. IEEE Transactions on Power Systems, Vol. 14, No. 4. 1999, pp. 1342-1348.

• [2] Baldick, R., A Generalized Unit Commitment Model, IEEE Transactions on Power Systems, Vol. 10, No. 1. 1995, pp. 465-475.

• [3] Teisberg, T.J., R.F. Weiher, and A. Khotanzad, The Economic Value of Temperature Forecasts In Electricity Generation, Bulletin of the American Meteorological Society, Vol. 86, No. 12, 2005, pp. 1765-1771.