EvergreenEcon.com

ESA 2011 Impact EvaluationDraft Report Public Workshop #2

August 7, 2013

Presented By: Steve Grover, President

2

Agenda for Today

Introductions (10 min)

Presentation of Draft Evaluation Report (30-45 min)

Q&A / Discussion (~2 hours)

Wrap Up / Next Steps (5 min)

Evaluation Team

The ESA Impact Evaluation team consists of the following:

Evergreen Economics (Prime Contractor)

Michaels Energy (Engineering Support)

CIC Research (Phone Surveys)

Betsy Wilkins (Technical Editor)

John Stevenson (Survey Design)

3

ESA Impact Evaluation Tasks

Task 1: Project Management

Task 2: Kickoff Meeting

Task 3: Research Plan

Task 4: Public Workshop #1

Task 5: Data Collection and Analysis

Task 6: Draft Report

Task 7: Public Workshop #2

Task 8: Final Report

Task 9: Data Documentation

4

Presentation Topics

The Draft Report Public Workshop presentation will focus on:

1. Impact estimates – methods and results

2. Phone survey findings

3. Evaluation conclusions and recommendations

Draft Report is available on the CPUC Energy Division website www.energywebdata.com/cpuc

Public comment period ends: August 12

Final Report due: August 31

5

Impact Methods: Billing Regression Model

Primary analysis method is a fixed effects billing regression model

Fixed effects model advantages:

•Fixed effects specification contains the desirable statistical properties for producing unbiased estimates

•Allows for controlling some of the non-program influences that affect savings through the use of dummy variables

•Consistent with prior ESA Program impact evaluations

6

Fixed Effects Model (kWh)

7

Fixed Effects Model (therms)

8

Model Variations

Three models estimated and included in the draft report:

1.Basic Model – includes all measures, uses entire sample

2.Measure Model – same model run separately for individual measures, uses sub-sample that installed each individual measure

3.Whole House Model – estimates total savings at the household level (separate models by house type, no breakdown by measure)

9

Savings Assignments

Impacts assigned at the measure level based on the following algorithm:

1. If the 95 percent confidence interval of the impact estimate from the Basic Model includes the ex ante savings value, then the estimate from the Basic Model is used.

2. If the confidence interval for Basic Model estimate did not include the ex ante value, then evaluator judgment was used to assign an impact value from among the Basic Model, Measure Model, or ex ante values.

3. In a couple of instances, an engineering estimate was assigned when the ex ante values were unusually high and neither of the regression models could provide a more reasonable result.

10

Data Screening for Regression Models

Screens used were very minimal, with observations deleted based on the following criteria:

•Master metered accounts

•Monthly observations > 10,000 kWh

•Monthly observations < 100 kWh

•Monthly observations > 5,000 therms

Screens resulted in very few observations or households being excluded from the analysis dataset (1-3%).

11

Data Screening (< 100 kWh and > 10,000 kWh)

12

Data Screening (> 5,000 therms)

13

Weather variable calculation

• Typical weather variable calculations for cooling-degree days (CDD) and heating-degree days (HDD) use daily averages for temperatures (measuring deviations from 65 degrees F).

• Traditional method will miss some heating/cooling activity – there may be some brief periods of heating or cooling, even if daily average does not meet 65 degree threshold.

• The current evaluation calculates weather variables based on hourly temperature deviations from 65 degrees, then averaged for entire day.

14

Weather variable comparison (CDD)

15

Weather variable comparison (HDD)

16

Impact Estimation Results

17

Household Savings Estimates

18

Impact Estimates Over Time (All housing types)

19

Whole House Model (Annual kWh Impacts)

20

Whole House Model (Annual Therm Impacts)

21

Post-Participation Energy Use / Pre-Participation Use

22

Factors Influencing Savings

Savings will vary across utilities and evaluation years for a variety of reasons:

•Weather

•Mix of measures installed

•Household energy usage

•Household demographics

•Home condition

•Regression model specification

23

Phone Survey Results

24

Phone Survey Overview

• Phone Survey sample = 150 participants per utility (600 total)

• Sample chosen from customers with a high increase (top 33% of increasers) in weather-normalized energy use before and after participating in the program.

• Questions focused on possible reasons for explaining increase in energy use

• Surveys fielded in April 2013 by CIC Research.

25

Phone Surveys Results: Cooling

26

Phone Survey Results: Heating

27

Phone Survey Results: Change in Occupancy

28

Conclusions

• Current impact estimates generally consistent with ex ante values

• Current estimates do deviate from prior impacts and DEER values, possibly due to multiple factors

• Whole House impact estimates consistently lower than those produced by assignments from Basic/Measure model.

• Consumers may be unaware that they are using more energy.

29

Recommendations

• Use hourly (rather than daily) method to calculate weather variables.

• If billing regressions used in future evaluations, having multiple models provides more flexibility on assigning final savings estimates.

• If wide variations in impacts is unacceptable, consider using deemed savings from DEER to provide stability.

• Allocate more time for evaluation, which will allow for more modeling options to be explored (including a multi-year analysis).

30

Remaining Project Timeline

Key Dates: Comments on Draft Report due: August 12, 2013

Final Report: August 31, 2013

Database and documentation delivery: September 2013

31