Home Electricity Report Program, January 2012 …...HOME ELECTRICITY REPORT PROGRAM JANUARY 2012...

HOME ELECTRICITY REPORT PROGRAM JANUARY 2012 THROUGH DECEMBER 2013 STUDY PERIOD

2013 Impact Evaluation Seattle City Light

Report No.: 1, Rev. 3

Date: July 2014

Project name: Home Electricity Report Program DNV GL Energy

SUS PAR

[Office Post 1]

[Office Post 2]

[Office Post 3]

[Office Post 4]

Tel: [+00 000 000 000]

[Enterprise No]

Report title: 2013 Impact Evaluation

Customer: Seattle City Light

Contact person: Sharon Noell

Date of issue: July 2014

Project No.: [Project No.]

Organization unit: [Name of unit]

Report No.: 1, Rev.3

Task and objective:

Prepared by: Verified by: Approved by:

Ken Agnew

[title]

[Name]

[title]

[Name]

[title]

Jon Farland

[title]

[Name]

[title]

Siyu Wu

[title]

[Name]

[title]

☐ Unrestricted distribution (internal and external) Keywords:

[Keywords] ☒ Unrestricted distribution within DNV GL

☐ Limited distribution within DNV GL after 3 years

☐ No distribution (confidential)

☐ Secret

Reference to part of this report which may lead to misinterpretation is not permissible.

Rev. No. Date Reason for Issue Prepared by Verified by Approved by

0 [yyyy-mm-dd] First issue

DNV GL – Report No. 1, Rev. 3 – www.dnvgl.com Page i

Table of contents

1 EXECUTIVE SUMMARY ..................................................................................................... 1

1.1 Evaluation Objectives and Overview 2

1.2 Key Findings 3

2 INTRODUCTION .............................................................................................................. 4

2.1 Program Description 5

2.2 Evaluation Objectives and Overview 6

2.3 Overview of This Report 7

3 RESEARCH DESIGN AND DATA COLLECTION ACTIVITIES ..................................................... 8

3.1 Experimental Design 8

3.2 Data Disposition 14

4 METHODOLOGY ............................................................................................................ 17

4.1 Fixed Effects Regression Model 17

4.2 Joint Savings Analysis 17

5 RESULTS ..................................................................................................................... 20

5.1 Program Savings 20

5.2 Measured Savings 21

5.3 Joint Savings 29

5.4 Rebate Program Joint Savings 29

5.5 Upstream Retail Lighting Program Joint Savings 32

6 CONCLUSIONS ............................................................................................................. 33

6.1 Recommendations 33

7 APPENDIX .................................................................................................................... 34

7.1 Tabular Measured Savings 34

7.2 Joint Savings Plots 38

7.3 Monthly Attrition 41

Figure 3-1: Initial Control / Treatment Splits by Zip Code ...................................................................... 8 Figure 3-2: Initial Experimental Design Schematic ................................................................................ 9 Figure 3-3: Final Control / Treatment Splits by Zip Code ...................................................................... 10 Figure 3-4: Final Experimental Design Schematic ................................................................................ 11 Figure 3-5: Pre-Program Randomization Test - Legacy Wave Treatment n=17,087, Control n=17,043 ...... 12 Figure 3-6: Pre-Program Randomization Test - Legacy CPW Wave Treatment n=5,444, Control n=417 ..... 13 Figure 3-7: Pre-Program Randomization Test - Expansion CPW Wave Treatment n=6,470, Control n=4,410

.................................................................................................................................................... 13 Figure 3-8: Pre-Program Randomization Test - Expansion Wave Treatment n=22,821, Control n=9,816 .... 14 Figure 5-1: Monthly per Household Savings Estimates (kWh)- Legacy Wave ........................................... 22 Figure 5-2: Monthly Overall Energy Saved (kWh) - Legacy Wave .......................................................... 23 Figure 5-3: Monthly Percent Savings – Legacy Wave ........................................................................... 23 Figure 5-4: Monthly per Household Savings Estimates (kWh) - Legacy CPW Wave ................................. 24 Figure 5-5: Monthly Overall Energy Savings (kWh) - Legacy CPW Wave ............................................... 25 Figure 5-6: Monthly Percent Savings - Legacy CPW Wave .................................................................... 25 Figure 5-7: Monthly per Household Savings Estimates (kWh) - Expansion Wave ..................................... 26

DNV GL – Report No. 1, Rev. 3 – www.dnvgl.com Page ii

Figure 5-8: Monthly Overall Energy Savings (kWh) - Expansion Wave ................................................... 27 Figure 5-9: Monthly Percent Savings - Expansion Wave ....................................................................... 27 Figure 5-10: Monthly per Household Savings Estimate per Household (kWh) – Expansion CPW Wave ........ 28 Figure 5-11: Overall Monthly Energy Saved (kWh) – Expansion CPW Wave ............................................ 28 Figure 5-12: Monthly Percent Savings - Expansion CPW Wave .............................................................. 29 Figure 5-13: Monthly Tracked Rebate Program Joint Savings Per Household ........................................... 30 Figure 5-14: Average Monthly Control and Treatment Rebate Savings for the Legacy Wave ..................... 31 Figure 5-15: Monthly Joint Savings for the Legacy Wave ...................................................................... 31 Figure 7-1: Average Monthly Control and Treatment Rebate Savings – Expansion CPW ........................... 38 Figure 7-2: Monthly Joint Savings – Expansion CPW ............................................................................ 38 Figure 7-3: Average Monthly Control and Treatment Rebate Savings – Legacy CPW ................................ 39 Figure 7-4: Monthly Joint Savings – Legacy CPW ................................................................................ 39 Figure 7-5: Average Monthly Control and Treatment Rebate Savings - Expansion ................................... 40 Figure 7-6: Monthly Joint Savings – Expansion ................................................................................... 40 Figure 7-7: Monthly Move Outs - Legacy Wave ................................................................................... 41 Figure 7-8: Monthly Move Outs - Legacy CPW Wave ............................................................................ 42 Figure 7-9: Monthly Move Outs - Expansion Wave............................................................................... 43 Figure 7-10: Monthly Move Outs - Expansion CPW Wave ..................................................................... 44 Table 1-1: SCL HER Program Wave Initial Populations ........................................................................... 2 Table 1-2: Summary of Annual Savings for SCL HER 2013 ..................................................................... 4 Table 2-1: SCL HER Program Wave Initial Populations ........................................................................... 6 Table 3-1: Billing Data Disposition - Legacy Wave ............................................................................... 15 Table 3-2: Billing Data Disposition - Expansion Waves ......................................................................... 15 Table 3-3: Household Attrition by Legacy Wave .................................................................................. 16 Table 3-4: Household Attrition by Expansion Wave.............................................................................. 16 Table 4-1: PSE Upstream Joint Savings Estimates .............................................................................. 19 Table 5-1: Seattle City Light HER Program savings, 2012 and 2013 ..................................................... 21 Table 5-2: Overall Measured Savings - Legacy Wave ........................................................................... 22 Table 5-3: Overall Measured Savings – Legacy CPW Wave ................................................................... 24 Table 5-4: Overall Measured Savings – Expansion Wave ...................................................................... 26 Table 5-5: Overall Measured Savings - Expansion CPW Wave ............................................................... 28 Table 5-6: Overall Annual Joint Savings Results and Precisions ............................................................. 32 Table 7-1: Monthly Measured Savings - Legacy Wave .......................................................................... 34 Table 7-2: Monthly Measured Savings - CPW Expansion ...................................................................... 36 Table 7-3: Monthly Measured Savings – Non-CPW Expansion ............................................................... 37

DNV GL – Report No. 1, Rev. 3 www.dnvgl.com Page 1

1 EXECUTIVE SUMMARY

Seattle City Light (SCL), a department of the City of Seattle, is one of the nation’s largest municipally owned

utilities in terms of the number of customers served. The department, which was established in 1910, is

managed by Seattle’s Mayor and overseen by the nine-person Seattle City Council. It owns significant

hydroelectric resources and its 131 square mile service territory includes Seattle and several surrounding

cities. SCL serves over 360,000 residential customers and almost 40,000 non-residential customers. In

2012, SCL provided its customers with 9.5 billion Megawatt-Hours of energy.

Seattle City Light operates the nation’s longest-running energy efficiency (EE) program. Currently, the EE

program is delivered by the Conservation Resources Division (CRD) to provide customers with various

energy efficiency services. CRD combines direct delivery and 3rd party delivery of a wide menu of EE choices

to its customers. These services result in a cost-effective energy resource for the utility that reduces

customer bills; avoids greenhouse gas emissions; and provides other positive environmental impacts.

Seattle City Light Conservation Resources Division launched the Home Electricity Report (HER) Program in

October, 2009. This program is designed to provide information to residential customers intended to

motivate changes by the customer that result in reduced billed energy consumption To enable this program,

SCL/CRD issued an RFP for a third party to clearly describe and deliver a program with goals to: 1)

Develop customized home energy reports based on SCL billing data to cost-effectively acquire sustainable

energy savings from the residential customer sector; 2) Design a test and control structure for the program

that will enable conclusive understanding of program efficacy over time; and 3) Provide SCL with semi-

annual reports to verify program accomplishments including change in electric consumption among those

customers receiving their home energy reports in comparison to a control group and to their historic

consumption. The winning bidder to CRD’s program implementation RFP was Opower Company whose

reports help customers reduce overall and peak electricity consumption via personalized report to customers

that use norm-based messages, provide energy efficiency tips and/or offers from SCL.

As agreed with Opower, the HER Program combines useful information that enables customers to make

informed choices about managing their energy costs. Several combinations of information can be provided

to each participant in each brief report. The household receives personalized information about their own

kWh consumption, along with information about equipment purchase or use; and about SCL/CRD EE

Programs. Each household also receives information that compares the subject household to a large group

of neighboring households.

A new report is sent to the treatment group subjects every two months shortly after receiving their bi-

monthly bill. This type of information presentation is thought to motivate households to use less energy and

to improve persistence of the related energy savings.

SCL’s HER program operational goals were to: 1) Achieve 12 million kWh of annual electricity energy

savings through providing normative information on electrical usage and targeted tips on energy saving

behaviors; 2) Increase participation in other SCL energy efficiency programs through program promotions;

and 3) Inform future decisions about program improvement, continuation or expansion of the HER

behavioral program.

Over the past 4 years, SCL sponsored two waves of HER program groups. The first wave, now referred to

as the Legacy wave was launched in October 2009. Households eligible for the first wave had at least one

year of billing history data and were in the top three quartiles of SCL customers in terms of annual electric


consumption. A second wave, referred to as the Expansion wave, was launched in July, 2011. Each wave

consisted of two groups into which households were randomly assigned: A treatment group that received

the HER program reports and a control group that did not. This experimental design is a key feature of the

HER program that facilitates precise, un-biased estimates of consumption impacts that are attributable to

the reports. Table 1-1 provides the counts of households in the treatment and control groups of the two

program waves.

Table 1-1: SCL HER Program Wave Initial Populations

Waves Treatment Control Total

Legacy Wave (from October, 2009) 22,531 17,460 39,991

Expansion Wave

(from July, 2011) 29,291 14,226 43,517

The expansion wave had two components. The first expansion group component paralleled the original

legacy group and had the same inclusion criteria. The second expansion group was conducted in co-

ordination with a City of Seattle single family weatherization program, called Community Power Works

(CPW). This CPW program was geographically based and the HER program focus included all CPW

participants regardless of consumption levels. The addition of the CPW program added complexity to the

evaluation design because many CPW households were already included in either the treatment or control

group of the Legacy wave. All CPW households were combined and received similar CPW-specific reports.

In addition, some of the legacy group members in the control group that resided in the CPW area were re-

assigned (randomly) to the CPW condition. Ultimately, savings were estimated for 4 groups rather than two

(Legacy, Legacy-CPW, Expansion, Expansion-CPW). These changes are detailed in the main report. The

changes did not compromise the validity of the estimated savings.

1.1 Evaluation Objectives and Overview

This impact evaluation quantifies the energy savings attributable to the HER program for 2012 and 2013

calendar years.

The primary objectives for the evaluation were the following:

1. Develop annual kWh savings impacts of 2012 and 2013 for the HER Program that comply with

Initiative 937 (RCW 19.285) and the associated Washington Administrative Codes; primarily 194-37-

080 (2)(a) which states that if a conservation measure life is less than two years, the utility can

claim the energy savings if it can verify that it has acquired the conservation for the entire biennium.

At this time, Seattle City Light treats savings from this program as having a measure life of one year

and must verify that the energy savings was acquired over the 2012-2013 time period.

2. Develop estimates of savings for four different program groups.

3. Quantify the extent to which the HER Program enhanced efforts of other SCL energy efficiency

programs and adjust HER program savings to account for any potential double counting or joint

savings.

4. Calculate a final credited savings amount that will meet Initiative 937 requirements for energy

savings.


To fulfill these evaluation objectives, billing and program participation data were included in a fixed effects

regression model which estimated consumption differences between treatment and control groups and

allowed for those differences to be evaluated for statistical significance. The regression model gave average

site-level savings results which could be aggregated into overall program savings. The site level regression

results were reduced by other kWh savings actions known to be made by the study household during the

study period. The site specific savings were reduced to account for these joint savings and ensure that any

program savings were only claimed once. The methods used here are consistent with industry best

practices for evaluation of these kinds of programs. Given the program experimental design and the

evaluation approach the results presented here are of the highest validity possible for any energy efficiency

program.

1.2 Key Findings

The main objective of this evaluation was to enumerate the savings uniquely associated with SCL’s HER

program. Table 1-2 presents the 2012 and 2013 savings estimates by wave and year as well as in total.

For this study, measured savings are based on the difference between observed consumption for treatment

households and observed consumption for control group households in the pre- and post-report periods.

Within measured savings, HER savings are the savings attributable to the HER program and joint savings,

which are attributed to the HER program and other SCL EE programs. Thus, for this study,

Measured Savings = HER Savings + Joint Savings

It is important to count savings only once and it is generally more convenient for joint savings to remain

with the other, non-HER programs. As a result, the HER program credited savings are the measured

savings less joint savings for both rebate programs and the upstream lighting programs. The HER program

credited savings are expressed at both the per household level and aggregated to reflect the remaining

active population of treatment households.

HER Credited Savings = Measured Savings -Joint Savings

The SCL HER program saved over 15 million kWh each year for 2012 and 2013, for a total of over 32 million

kWh for the two years combined.1

1 These savings are site savings at the meter, not busbar savings.


Table 1-2: Summary of Annual Savings for SCL HER 20132

Recipient Wave Year

Measured Savings

(kWh per HH)

Joint Savings (kWh per HH)

Credited Savings

Rebate Upstream

kWh Per HH

Full Year Recipient

Households

All Households

(kWh)

Legacy 2012 422.6 (+/- 79) 0 2.3 420.3 14,311 6,015,259

2013 425.1 (+/- 113) 0 5.5 419.7 13,478 5,656,026

Legacy

CPW

2012 534.6 (+/- 350) 0 2.3 532.3 4,960 2,640,304

2013 640.6 (+/- 477) 0 5.5 635.2 4,679 2,971,623

Expansion 2012 254.5 (+/- 64) 0 0.9 253.7 20,876 5,295,244

2013 333.4 (+/- 113) 0 1.6 331.8 19,422 6,445,177

Expansion

CPW

2012 250.2 (+/- 95) 0 0.9 249.3 5,902 1,471,703

2013 307.5 (+/- 119) 0 1.6 305.9 5,476 1,674,988

Total 2012 15,422,510

2013 16,747,814 Grand

Total 32,170,323

Opower also reported their own estimates of savings for the SCL HER programs. Their estimates,

13,443,050 and 14,534,931 kWh for 2012 and 2013, respectively, were similar but slightly below DNV GL’s

estimates for those years. The most likely explanation for the difference concerns the calculation of savings

for the Legacy CPW group which was restarted in 2011. The way Opower accounted for this change in group

membership was unnecessarily conservative and, as a result, underestimated savings.

2 INTRODUCTION

Seattle City Light (SCL), a department of the City of Seattle, is one of the nation’s largest municipally owned

utilities in terms of the number of customers served. The department, which was established in 1910, is

managed by Seattle’s Mayor and overseen by the nine-person Seattle City Council. It owns significant

hydroelectric resources and its 131 square mile service territory includes Seattle and several surrounding

cities. SCL serves over 360,000 residential customers and almost 40,000 non-residential customers. In

2012, SCL provided its customers with 9.5 billion Megawatt-Hours of energy. SCL is supported by revenues

from its customers and has total operating revenues of over $800 million.

Seattle City Light has operated the nation’s longest-running energy efficiency program. Since 1977, the

Conservation Resources Division (CRD) has provided its customers energy efficiency services. The results of

these efforts have been significant: delivering a cost-effective energy resource for the utility; reducing

residential, commercial and industrial customers’ bills; and avoiding greenhouse gas emissions and other

environmental impacts of energy production.

Seattle City Light’s Conservation Resources launched the Home Electricity Report (HER) Program in October,

2009. The initial wave of the program has continued to the present time, while additional households were

added to the program in 2011. This evaluation establishes the savings produced by all waves of the

2 Confidence level is 95%.


program since the program’s inception. Particular focus for this report is 2012 and 2013 program savings

for the purpose of meeting Washington State’s Initiative 937 requirements for energy savings.

2.1 Program Description

The HER Program utilizes enhanced billing information and social marketing tools to encourage responsible

energy behavior and choices. The reports provide recipients with feedback on their household energy use

along with a comparison of the recipient household’s energy usage with that of neighboring homes. These

reports were mailed out on a bi-monthly basis, timed to come out shortly after customers received their

electric utility bills. Neighboring homes were described as approximately 100 occupied, nearby homes that

are similar in size to the recipient’s home. This social norming approach uses a form of peer pressure to

motivate energy savings. In addition, the reports provide practical tips regarding steps households can take

to reduce energy consumption through behavioral changes and participation in other SCL energy efficiency

programs. The program operational goals were to: 1) Achieve 12 million kWh of annual electricity energy

savings through providing normative information on electrical usage and targeted tips on energy saving

behaviors; 2) Increase participation in other SCL energy efficiency programs through program promotions ;

and 3) Inform future decisions about program improvement, continuation or expansion of the HER

behavioral program.

The initial HER Program wave, started in October, 2009, consisted of approximately 40,000 households with

about 22,000 receiving home energy reports and about 17,000 not receiving reports. Households eligible

for the first wave had at least one year of billing history data and were in the top three quartiles of SCL

customers in terms of annual electric consumption.

In July of 2011, Seattle City Light’s Conservation Division (CRD) launched an expansion of the original HER

population that included two sub-groups. One group targeted participants in another SCL program,

Community Power Works (CPW), which was a residential weatherization program that was conducted in a

specific geographic area of Seattle. Households from this program (who were not already included in the

Legacy program) were selected regardless of electric consumption levels, in contrast to the other groups

where the lowest 25% of electric energy consumers were excluded. An expansion group targeted an

additional 20,000 households from the general SCL service area population with the same emphasis on high

consumption households as the Legacy wave.

The HER Program waves were all organized using an experimental design to facilitate the evaluation of

customer savings. For each program, the households that received reports were a randomly assigned

subset of a larger group of eligible customers. In this way, a group of similar households were held out as a

“control” group to represent customers who did not receive the reports. Table 2-1 provides the overall

counts of customers included in the treatment (report recipient) and control (non-recipient) groups.3

3 The legacy wave counts reported here represent the final allocation of the Legacy group. The initial treatment group was actually very close 20,000

customers but additional households were moved (on a randomized basis) from the control to treatment groups at the time of the start of the

expansion wave. This issue is discussed at length in section 3.1.


Table 2-1: SCL HER Program Wave Initial Populations

Waves Treatment Control Total

Legacy Wave (from October, 2009) 22,531 17,460 39,991

Expansion Wave (from July, 2011) 29,291 14,226 43,517

In total, 51,822 households received reports at some time since October, 2009. With natural attrition due to

changes in occupancy, the total number of households still receiving the reports was 41,703 at the end of

2013.

The Randomized controlled trial (RCT) organization is the gold standard for quantitative evaluation of

program-related change and identical to the technique accepted for use in medical and pharmaceutical

experiments. These groups were randomly assigned in advance of the first reports to establish their

similarity in the most rigorous possible fashion. The results from an RCT experimental design meet the

highest standards of validity. In addition to providing an un-biased estimate of savings, the large number of

participating households supports results with extremely high statistical precision. Savings estimates from

programs organized in this way represent the most robust and rigorous energy evaluation results that are

produced today.

2.2 Evaluation Objectives and Overview

1. Develop an overall estimate of 2012 and 2013 HER Program savings that will comply with Initiative

937 (RCW 19.285) and the associated Washington Administrative Codes; primarily 194-37-080 (2)(a)

which states that if a conservation measure life is less than two years, the utility can claim the

energy savings if it can verify that it has acquired the conservation for the entire biennium. Given

the nature of this program, Seattle City Light believes the measure life is one year and would need

to verify that the energy savings have been acquired over the 2012-2013 time period.

2. Develop individual estimates of savings for four different program groups.

3. Quantify the extent to which the HER Program enhanced efforts of other SCL energy efficiency

programs and adjust HER program savings to account for any potential double counting or joint

savings.

4. Calculate a final credited savings amount that will meet Initiative 937 requirements for energy

savings.

To fulfil these evaluation objectives, billing and program participation data were include in a fixed effects

regression model which estimated consumption differences between treatment and control groups and

allowed for those differences to be evaluated for statistical significance. The regression model gave average

site-level savings results which could be aggregated into overall program savings. Joint savings were

calculated use energy efficiency program tracking data and secondary sources. The site specific savings

were reduced to account for the joint savings and guarantee any program savings were only claimed once.

The methods used here are consistent with industry best practices for evaluation of these kinds of programs.

Given the program experimental design and the evaluation approach the results presented here are of the

highest validity possible for any energy efficiency program.


2.3 Overview of This Report

This report discusses the efforts, methods and results that drove this impact evaluation. It is organized as

follows: Section 1 is the report Executive Summary. Section 2 is the report Introduction. Section 3 of this

report discusses the overall study design and relevant data, Section 4 describes the methodology used to

calculate savings estimates, Section 5 presents the program impact results for the program years 2012 and

2013, while Section 6 offers conclusions and recommendations.


3 RESEARCH DESIGN AND DATA COLLECTION ACTIVITIES

3.1 Experimental Design

The evaluation of a HER program takes full advantage of randomized controlled trial experimental design.

Ideally, the experimental design allows for optimal evaluation after the fact. Opower, the implementer of

this HER program has an established track record of creating sound experimental designs for its programs.

The design for the SCL HER program was more complicated than a typical Opower HER program

experimental design. Opower initially indicated that there were, in fact, three separate RCT experimental

designs: the legacy wave, an expansion wave of the remaining households in the Community Power Works

program and a further expansion wave of general eligible households. Figure 3-1 illustrates the location and

population proportions of those three waves.

Figure 3-1: Initial Control / Treatment Splits by Zip Code

These maps explain a number of aspects of the original experimental designs. Most of the Legacy wave

(first panel) was allocated to treatment and control at a 50% rate. The CPW and expansion waves (right

two panels) were mostly geographically distinct. The Community Power Works (CPW in the second panel)

allocation captured the majority of the remaining eligible households in that geographical area. The

Expansion wave (third panel) allocation pulled from the remaining areas. The later waves were allocated

with a greater percentage of the eligible population going to the treatment group (approximately 70%). As

shown in the more recent groups were more unevenly allocated between Treatment and Control group

assignment.

Opower indicated that the expansion CPW group was designed to include all CPW program households that

were not included in the initial legacy wave. In addition, Opower indicated that the CPW subset from the

Legacy wave and the Expansion CPW wave were effectively combined once the Expansion waves were


started. This meant that the CPW households received information that was specifically appropriate to the

CPW program for which they were all eligible given their geographic location.

Figure 3-2 illustrates the organization of the experimental design as related by Opower. In mid-2011,

Opower divided the Legacy treatment and control groups into two based on the primarily geographical CPW

eligibility area. Because the split was geographical, the initial randomized assignment to treatment and

control is maintained within such a subset. Opower combined the CPW subset of the Legacy wave with the

remainder of CPW households which were already separately and randomly assigned. In the figure, the

T(CPW) and C(CPW) portions of the Legacy wave combined with the CPW population making two newly

defined waves, a reduced Legacy wave and a complete CPW wave.

Figure 3-2: Initial Experimental Design Schematic

This explanation of the HER program experimental design explained all but one aspect of the group

membership observed among the eligible population. The uneven allocation of the Legacy group to

treatment and control remained a concern that Opower was unable to explain. Referring again to Figure 3-1

above, the Legacy wave allocations (pie charts) in the CPW geographical area are clearly different than the

remainder of the Legacy wave. This differential allocation within the original experimental design would

have implications for the evaluation even if these areas were allocated randomly to these proportions within

these areas. It is essential that all control group households have the same effective weight in the analysis.

It is similar to a stratified design with differential allocations within strata. If the treatment and control

groups do not have equal allocation, the strata, in this case, zip code areas, would have to be weighted so

as to have equal weight across treatment and control.

A second map of the program populations brought clarity to the situation. Figure 3-3 splits the Legacy wave

into CPW households in the original Legacy wave and the remaining non-CPW households. The effect on the

treatment-control allocations among Legacy wave households is dramatic. The non-CPW Legacy wave now

appears to be equally allocated across all zip codes. Similarly, the Legacy CPW wave appears to have a

consistent but much different allocation across the CPW zip codes. The data clearly indicate stratification on

the CPW identity


Figure 3-3: Final Control / Treatment Splits by Zip Code

Ultimately, with the help of Opower, we were able to uncover what occurred. Though it was not indicated in

the data provided, the Legacy group was, in fact, originally allocated on a 50-50 basis. This included any

households selected into that group that were in the area later designated as the CPW area. At the start of

the Legacy HER program in 2009 there were 2,909 CPW households within the treatment group and 2,952

CPW households in the control group.

When the CPW wave was constructed, Opower wanted to combine all CPW households together and

maximize the number of CPW household receiving reports. To do so, they not only separated the CPW

households from the original Legacy wave, they also re-allocated a large number of household that were

previously control group households not receiving reports and put them into a new treatment group

receiving special CPW reports. From that point on there were 5,444 households that were originally in the

Legacy wave that were now in the CPW treatment group and only 417 in the control. This is illustrated in

the Legacy CPW map in Figure 3-3.

DNV GL only received the treatment and control group memberships from the later iteration of the program.

We did not receive data that indicated the original 50-50 split or who was in the original control group and

then switched to the treatment group for the second wave.

Our solution, even prior to uncovering the full explanation, was to treat the original three waves as four

waves. The Legacy group was treated as two unique groups that were now referred to as the Legacy group

and the Legacy CPW group. The Legacy CPW treatment group was comprised of the Legacy control

households in the CPW area that were later randomly assigned to the CPW treatment condition as well the

Legacy treatment households that were in the CPW area. The Legacy CPW control group was comprised of

those in the Legacy control group who were in the CPW geographic area. Figure 3-4 illustrates how we


treated the final experiment design given the split of the Legacy population. Unlike Figure 3-2, the Legacy

population is treated as two separate populations from the beginning.

Figure 3-4: Final Experimental Design Schematic

This Legacy CPW organization in Figure 3-4 does not explicitly show the re-allocation performed in 2011.

Instead, to simplify the organization, those Legacy CPW households that started out in the control group and

only started receiving reports in the expansion period were included as treatment households for the whole

program period. This does not affect overall treatment group savings. Organized in this fashion, the

average per household savings might appear smaller during the first two years because it includes those

households that did not receive reports. However, the total savings is not affected because it is calculated by

assigning this smaller amount of savings to the larger number of treatment households only some of which

received reports or not.4

In fact, the approach we took to estimate the savings for the Legacy CPW households is a more accurate

measure of the full savings than the approach used by Opower. Our approach maintains the true pre-

program baseline for Legacy CPW households. In contrast, by “re-starting” the CPW wave during the

expansion period, the Legacy CPW treatment households had a new “pre-report” baseline period during

which many households received reports. This effectively re-set their consumption baseline lower and left

savings on the table.

The only potential downside to this approach is that the individual Legacy CPW savings estimates, when

calculated separately, have relatively low precision because the overall size of the wave is relatively small

and the control group, because of the disproportionate allocation, is extremely small. Ultimately, the results

for this group are still statistically significant on their own. Furthermore, it would also be possible to

combine them with the remaining Legacy group using appropriate weights and get a higher single program-

4 The strength of a RCT experimental design is that it will capture savings if all household are saving or only a subset are saving. For a treatment

group of 100 households, the average per household savings will be 50 kWh if all households save 50 kWh or if 10 of the 100 households save

500 kWh and the remainder saving nothing. In both cases, the total savings would be 5000 kWh. If, in the later scenario, you could identify

those that saved 500 kWh, you could say those 10 households had an average per household savings of 500 kWh. The total savings would still

be 5000 kWh because only those 10 houses had any savings.

In this case, during the first two years of the legacy CPW program, some treatment households were not actually receiving the reports. Despite

this, the overall savings estimate for this group is still valid. The difference is measured for all treatment group members whether they received

the report or not and that average difference is applied to all treatment group members.


level precision. We choose not to do that here to maintain full transparency and because minimum

precisions levels were met.

Despite the lower precision of the Legacy CPW group, the results are valid from an experimental design

perspective. In general, it is best to set up the experimental design and not change it. A random re-

allocation of control households to the treatment group makes the evaluation more complicated to develop

and explain. With respect to the overall validity of the results, what occurred with the Legacy CPW program

was reasonable because the change in group membership was randomized. This re-allocation can only

occur in one direction (from control to treatment) and this limits the extent to which reallocation can occur.

3.1.1 Randomization Tests

In order to insure the validity of this impact evaluation, statistical tests were conducted to gauge the

balance of the experimental design. If households were randomly assigned to control and treatment groups,

then we would expect to see no statistically significant difference in energy usage between the two groups in

the pre-program period. Overall, there is no statistical difference in pre-program consumption for any of the

four waves at 95% confidence.

DNV GL conducted these statistical tests at a bimonthly level consistent with the underlying billing data.

Each two month period’s consumption was isolated and tested for differences between treatment and control.

Figure 3-5 displays the results of the monthly testing for the Legacy wave. None of the differences between

treatment and control group consumption for any of the pre-report periods are statistically significantly

different than zero.

Figure 3-5: Pre-Program Randomization Test - Legacy Wave Treatment n=17,087, Control n=17,043


Figure 3-6, Figure 3-7 and Figure 3-8 present the same statistical test of differences applied to the Legacy

CPW, Expansion-CPW and Expansion waves, respectively. The confidence intervals vary in width depending

on the size of the treatment and control groups in the wave. In particular, the relatively small Legacy CPW

wave has wide confidence intervals in the pre-period.5 In all cases, the confidence intervals encompass zero.

Figure 3-6: Pre-Program Randomization Test - Legacy CPW Wave Treatment n=5,444, Control n=417

Figure 3-7: Pre-Program Randomization Test - Expansion CPW Wave Treatment n=6,470, Control n=4,410

5 Specific wave treatment and control counts are provided in next section in Table 3-1 and Table 3-2.


Figure 3-8: Pre-Program Randomization Test - Expansion Wave Treatment n=22,821, Control n=9,816

3.2 Data Disposition

This evaluation used relevant information such as customer consumption records as well as energy efficiency

program tracking data. The subsequent data were reviewed and assessed for quality.

3.2.1 Consumption Analysis

SCL provided consumption records for treatment and control members of each wave to Opower, the

program implementer, on a regular scheduled basis. The data were used by Opower for ongoing savings

estimates and to create the comparative information for the reports. Opower provided a complete data file

back to SCL for use in this analysis.

These data were billing meter reads which typically followed a 60-day read schedule. Sites that did not

meet at least one of the following criteria were flagged and removed from the model dataset.

Premises with less than six bi-monthly pre or post bill reads.

Premises with observations of zero or missing read dates usage.

Premises with greater than 400 kWh daily average usage for a given year.6

Because the read schedule of the billing data was approximately 60 days on average, a full calendar year is

approximately six billing meter reads. At the very minimum, a full year of consumption is required to

account for seasonality. For this reason, any premises with less than 12 months of pre- or post-program

consumption were dropped from the analysis.

6 This is an arbitrary cut-off that is greater than 10 times the average program household consumption and removes a small number of households.


Eliminating data anomalies and outliers such as these is a common data treatment for these types of studies.

These data preparation and cleaning treatments do not bias results because they are applied to both

treatment and control groups and are uncorrelated with the random group assignment. Further, for this

evaluation study, the excluded sites were approximately even across treatment and control members across

all waves and only constituted a small percentage of the overall study group. The model results are still

applied to the full population of HER treatment households for that calculation of savings. The underlying

assumption for a data removal of this kind is that the remaining sites are representative of the removed

sites.

Table 3-1 displays the counts of excluded households for the Legacy wave and overlapping households,

while Table 3-2 displays the dropped households indicative of the Expansion wave.

Table 3-1: Billing Data Disposition - Legacy Wave

Population

Legacy Legacy CPW

Treatment n

% Removed

Control n

% Removed

Treat-ment

n

% Removed

Control n

% Removed

Original population

17,087 17,043 5,444 417

Insufficient Pre

214 1.3% 219 1.3% 0 0.0% 0 0.0%

Insufficient Post

230 1.3% 232 1.4% 1 0.0% 0 0.0%

Zero Usage 217 1.3% 219 1.3% 1 0.0% 0 0.0%

High Usage 83 0.5% 99 0.6% 19 0.3% 2 0.5%

Final Sample 16,343 4.4% 16,274 4.5% 5,423 0.4% 415 0.5%

Note: Some sites have multiple issues.

Table 3-2: Billing Data Disposition - Expansion Waves

Population

Expansion CPW Expansion

Treatment n

% Loss Control

n % Loss

Treatment n

% Loss Control

n % Loss

Original population

6,470

4,410 22,821 9,816

Insufficient Pre 165 2.6% 108 2.4% 843 3.7% 350 3.6%

Insufficient Post 134 2.1% 92 2.1% 499 2.2% 190 1.9%

Zero Usage 128 2.0% 83 1.9% 477 2.1% 184 1.9%

High Usage 14 0.2% 8 0.2% 60 0.3% 33 0.3%

Final Sample 6,029 6.9% 4,119 6.6% 20,942 8.2% 9,059 7.7%

Note: Some sites have multiple issues.

Table 3-3 summarizes the counts of households with respect to natural occupancy attrition. The level of

attrition seen here is consistent with typical move-out patterns. The first year numbers for each wave

represent the attrition in that partial year. The Legacy CPW group shows little or no attrition during the first

two years of the program because it was defined in 2011 based on who was still present in the program.


Table 3-3: Household Attrition by Legacy Wave

TimeFrame

Legacy Legacy CPW

Treatment Control Treatment Control

Counts Attrition Counts Attrition Counts Attrition Counts Attrition

Program Start 17,087 17,043 5,444 417

End of 2009* 16,789 1.7% 16,771 1.6% 5,444 0.0% 417 0.0%

End of 2010 15,638 6.9% 15,585 7.1% 5,434 0.2% 416 0.2%

End of 2011 14,698 6.0% 14,667 5.9% 5,119 5.8% 385 7.5%

End of 2012 13,849 5.8% 13,777 6.1% 4,805 6.1% 367 4.7%

End of 2013 13,084 5.5% 13,085 5.0% 4,561 5.1% 350 4.6%

Overall Attrition 23.4% 23.2% 16.2% 16.1%

*Partial year. Legacy program started in October

Table 3-4: Household Attrition by Expansion Wave

TimeFrame

Expansion CPW Expansion

Treatment Control Treatment Control

Counts Attrition Counts Attrition Counts Attrition Counts Attrition

Expansion Start 6,470 4,410 22,821 9,816

End of 2011* 6,137 5.1% 4,190 5.0% 21,728 4.8% 9,324 5.0%

End of 2012 5,656 7.8% 3,858 7.9% 20,095 7.5% 8,666 7.1%

End of 2013 5,275 6.7% 3,582 7.2% 18,783 6.5% 8,130 6.2%

Overall Attrition 18% 19% 18% 17% * Partial year. Expansion programs started in July.

Tables 3-3 and 3-4 provide the household counts at the end of each calendar year. In practice, the savings

regression models include data for a household up until the point they move and go inactive. Thus, in the

regression analysis, household counts decrease each month due to new attrition. Savings are calculated by

combining monthly savings with the number of households still active during that month. The “full year

recipient” counts used to estimate final total savings (in Table 1-2, above and Table 5-1, below) reflect the

number of effective full year recipients given this natural monthly attrition. The number of full year

recipients falls somewhere between the counts at the start and end of the year and will vary based on the

rate of attrition through the year.

3.2.2 Program Tracking Data

DNV GL received extensive rebate program tracking data from SCL. These data were carefully vetted by

SCL staff prior to being passed to us. We integrated these data and applied savings, where not included,

based on SCL deemed savings. Aggregate savings were calculated and compared to overall savings to make

sure savings allocations were correct.


4 METHODOLOGY

This evaluation used bi-monthly electricity consumption data as well as downstream rebate tracking data to

quantify the savings attributable to the HER program.

4.1 Fixed Effects Regression Model

For a behavioral program such as the Home Electricity Reports, measuring the reductions in electricity

consumption is most appropriately done using a fixed-effects regression model. For the current context, the

fixed-effects regression provides a difference of differences calculation from pre- to post-program and from

treatment to control. The change in treatment consumption from pre-program to post- is made relative to

the same change in control consumption. Thus comparable differences can be statistically calculated.

The difference of differences analysis in a regression context facilitates several advantages that otherwise

might confound the savings estimates. Specifically, non-program change in electricity usage patterns across

households and across time is explicitly accounted for in the model. Each household and each time period is

treated as an average “fixed” effect relative to all others. The fixed effects model can be fit on a monthly,

annual, or overall pre / post time frame and all models capture the same treatment effect.

The fixed effects model for each wave is:

Where:

= Average daily usage of customer i during period t

= Interaction term denoting a household is a treatment in post period j7

= Regression coefficient associated with post period j

= Month/year fixed effect.

= Premise-level fixed effect.

= Regression disturbance.

Monthly measured savings estimates are derived from the parameters . These regression coefficients

measure the energy impacts of being in the treatment group in the post period. The term controls for site-

specific effects that vary across households, but not over time. Conversely, the term controls for time-

specific effects that do not vary across households. This approach is consistent with the best practices put

forth in the “State and Local Energy Efficiency Action Network’s Evaluation, Measurement, and Verification

(EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations8

4.2 Joint Savings Analysis

The joint savings analysis is required to guarantee that the savings claimed by the HER program are not also

claimed by some other SCL energy efficiency program. The consumption analysis at the heart of this

7 The subscript j is a time period and is a subset of the subscript i. We use j as a means to delineate between time periods in the post period only

(e.g., j = 1, 2, …, J) and time periods that span the entire analysis range (t = 1, 2,…, T). 8 State and Local Energy Efficiency Action Network. 2012. Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy

Efficiency Programs: Issues and Recommendations. Prepared by A. Todd, E. Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National

Laboratory. http://behavioranalytics.lbl.gov.


evaluation, measures all savings that are observed at the household that are caused by the HER program.

Most of the savings are caused by the HER Program alone and those savings are reflected in the final

“credited” savings numbers this evaluation reports. It is also possible that some of the savings captured in

the consumption analysis were the result of the HER Program working jointly with other SCL rebate

programs.

The HER program does, in fact, promote the other programs available to SCL customers, so it would not be

unexpected to find that HER program treatment group members were more likely to take advantage of

these programs. Joint savings only happen because both programs were present. Despite the fact that

joint savings are produced by multiple programs, each kWh saved should only be counted by one of the

programs. The easiest way to do that accounting is to net the joint savings from the additional program

activity out of the measured HER program consumption reduction or measured savings. The joint savings

analysis quantifies the magnitude of these savings that are jointly shared by the HER program and other

energy efficiency programs.

4.2.1 Downstream Rebate Programs

Energy efficiency purchases that occur directly through SCL rebate programs are tracked in SCL data

systems. DNV GL analyzed SCL rebate program tracking data to identify the possible increased uptake of

these other SCL energy efficiency programs by the treatment groups relative to the control group. These

programs include clothes washers, energy efficient space and water heating systems, etc. In these program

tracking data systems, rebate program participation and associated savings are tied directly to the customer

within the HER program treatment and control groups. The experimental design framework makes it

possible to accurately measure any increased activity in other energy efficiency programs by the HER

treatment groups.

For this analysis, DNV GL compiled data on all rebated installations in the post-report periods for both

treatment and control groups. Savings were assigned on an average daily basis starting with the installation

date and carrying forward. Savings are apportioned equally across the days after installation on an average

daily savings basis. Savings are assumed to stay in place through the measure’s useful life.9 For each

year’s rebate program joint savings calculation, the average per household accumulated savings of the

control group for that year are removed from the average per household accumulated savings of the

treatment group for that year. Any increase in savings for the treatment group is recognized as the effect of

the HER program on rebate program activity.

Any additional activity in a downstream rebate program, or joint savings, by HER households during the HER

period of study represents kWh savings that might erroneously be attributed to both the rebate and the HER

programs. The most common way to address this concern is to remove these joint savings from the HER

program measured savings. Thus, the credited HER savings are the difference between measured and joint

savings. In aggregate,

Credited HER savings = Measured savings – Joint Savings.

Removing joint savings from either the HER program savings or the original rebate program savings is

necessary to preclude double counting for increased energy efficiency program activity due to HER.

9 All measure lives are at least as long as the five years the HER Program has been in place.


4.2.2 Upstream Retail Lighting Joint Savings

The Twist and Save Program is an upstream program that reduces retail prices for energy efficient lights like

CFLs and, more recently, LEDs. The HER Program interacts with this kind of program in the same way as

they do with rebate programs. The HER program is expected to motivate increased purchases of bulbs

either due to the general conservation message or specific recommendations. As with rebate programs, if

additional bulbs were purchased and installed in the average treatment household compared to the average

control group household, the resulting additional savings would show up in the consumption analysis as HER

Program savings.

The Twist and Save program tracks savings based on sales of the bulbs and fixtures from participating

retailers. The retail program is not able to track purchases to individual households. Because of this, the

direct quantification of the joint savings through program tracking data is not feasible as with rebate

program savings. An alternative approach is to survey a sample of treatment and control households to

ascertain number of energy efficient (CFL and LED) bulb purchases. The sample-based estimate of

increased purchases translates into a parallel estimate of upstream program joint savings.

Careful estimation of the upstream program joints savings is not the norm in HER program evaluations.

Many evaluations ignore the potential for this kind of double counting or address the issue with extremely

small sample surveys that produce non-statistically significant results that are then ignored. SCL has the

advantage of having a leader in the area of upstream joint savings estimation as a neighboring utility.

Puget Sound Energy (PSE) has performed extensive survey work annually for the last three years and has

some of the best estimates of upstream joint savings in the country.10

For this evaluation, we leveraged the PSE upstream savings estimates produced for the 2012 PSE HER

Program Evaluation. These estimates cover upstream joint savings for a program that is similarly structured

to SCL’s Twist and Save efficient lighting discount program. The estimates cover the first four years of a

HER program in the same time period and geographical area so are ideally suited as estimates of the SCL

Twist and Save program-related joint savings. Table 4-1 provides the annual estimate of joint savings in

kWh for the PSE upstream program.

Table 4-1: PSE Upstream Joint Savings Estimates

Program Year Joint Savings

(kWh)

Year 1* 0.86

Year 2 1.59

Year 3 2.32

Year 4 5.47 * Includes last two months of 2008

For the Legacy waves which started in 2009, the 2012 and 2013 savings occur in the third and fourth

program year. We thus use years three and four of the PSE Upstream program joint savings estimate. For

10

Year 4 Home Energy Report Evaluation, Puget Sound Energy. DNV GL. https://conduitnw.org/pages/file.aspx?rid=1611


the expansion waves, we use years one and two of the PSE estimates. These estimates show a small but

consistent increase in upstream joint savings over the years of the analysis. The fact that the estimates are

quite small relative to the overall savings measured for the SCL HER Programs supports SCL’s decision to

not invest in a full set of joint savings surveys. On the other hand, it is appropriate to address this

possibility of double counting and net this estimate of joint savings out of the overall measure consumption

reduction. 11

Overall, HER savings can be expressed as Measured Savings less Rebate Savings and less Upstream Joint

Savings. HER savings = Measured savings – Rebate Savings – Upstream Joint Savings.

5 RESULTS

In this section, we present the savings estimates derived from consumption data. There are several terms

used to delineate between savings estimates.

- Measured Savings are the estimate of overall consumption reduction caused by the HER program.

These are calculated in a regression framework that compares pre- versus post-report period

consumption for both treatment and control groups. Within the randomized controlled trial

experimental design, the control group provides the best possible estimate of what treatment

households would have done in the absence of the program, making this a particularly accurate and

reliable estimate of overall program-related savings.

- Joint Savings are savings produced by the HER program in concert with other energy efficiency

programs at SCL. In the interest of avoiding double counting of savings, all of the known joint

savings are removed.

- Credited Savings are the final savings estimate of the HER program after the removal of the joint

savings. SCL’s HER program, and only the HER program, generated these savings. The

combination of the experimental design and the care with respect to potential double counting make

these results as robust as any results in energy program evaluation.

5.1 Program Savings

The key results for this evaluation are the 2012 and 2013 HER program savings for each of four

implemented waves. Table 5-1 provides these results. The measured savings results are annual, per

household estimates of savings. They range from over 400 kWh for the third and fourth years of the Legacy

wave to 250 to 300 kWh hours for the first and second years of the Expansion and Expansion CPW waves.

The small Legacy CPW wave has the highest per household savings, at greater than 500 kWh her household,

perhaps reflecting the increased emphasis on conservation accompanying the CPW program. All the

measured savings are statistically significantly different than zero at 95 percent confidence.

11

It is extremely difficult to get statistically significant estimates of the uplift in upstream program lighting sales. The PSE survey called

approximately 600 households in both the treatment and control groups each of the last two years to develop these estimates. Despite this

substantial sample size, the estimates were not statistically significant. Because they were based on such large samples and were based on the

RCT experimental design PSE decided the safe decision, with respect to risking double counting, was to net these joint savings out regardless of

the lack of statistical significance.


Table 5-1: Seattle City Light HER Program savings, 2012 and 201312

Recipient Wave Year

Measured Savings

(kWh per HH)

Joint Savings (kWh per HH)

Credited Savings

Rebate Upstream

kWh Per HH

Full Year Recipient

Households

All Households

(kWh)

Legacy 2012 422.6 (+/- 79) 0 2.3 420.3 14,311 6,015,259

2013 425.1 (+/- 113) 0 5.5 419.7 13,478 5,656,026

Legacy

CPW

2012 534.6 (+/- 350) 0 2.3 532.3 4,960 2,640,304

2013 640.6 (+/- 477) 0 5.5 635.2 4,679 2,971,623

Expansion 2012 254.5 (+/- 64) 0 0.9 253.7 20,876 5,295,244

2013 333.4 (+/- 113) 0 1.6 331.8 19,422 6,445,177

Expansion

CPW

2012 250.2 (+/- 95) 0 0.9 249.3 5,902 1,471,703

2013 307.5 (+/- 119) 0 1.6 305.9 5,476 1,674,988

Total 2012 15,422,510

2013 16,747,814 Grand

Total 32,170,323

The rebate and upstream program joint savings are provided by wave and year. There were no statistically

significant rebate program joint savings for any of these waves. Thus, there is no indication that the HER

program resulted in increased participation in other SCL rebate programs. Other evaluations of Northwest

utilities’ HER programs have also found no evidence of significant joint electric savings.13 Upstream program

joints savings, though not statistically significant, are still removed from measured savings to rigorously

avoid double-counting savings. Credited savings are provided both on a per household basis and total

savings for the active households for each wave and year. Combined HER programs savings are calculated

by year and overall. At the program level, 15,422,510 kWh were saved in 2012 and 16,747,814 kWh were

saved in 2013. Thus in general, total savings are over 15 million kWh for each year, for a total savings of

over 32 million kWh for the two year period14.

Opower also reported their own estimates of savings for the SCL HER programs. Their estimates,

13,443,050 and 14,534,931 kWh for 2012 and 2013, respectively, were similar but slightly below DNV GL’s

estimates for those years. The most likely explanation for the difference concerns the calculation of savings

for the Legacy CPW group which was restarted in 2011. The way Opower accounted for this change in group

membership underestimated savings.

5.2 Measured Savings

This section discusses the measured savings estimated from the electricity consumption of treatment and

control groups for all program waves. The year 2009 was a partial program year for the Legacy wave while

the year 2011 was a partial program year for the CPW and non-CPW expansion.

12

Confidence level is 95%. 13

The Puget Sound Energy HER program has never found statistically significant electric joint savings. 14

These savings are site savings at the meter, not transformer or busbar savings.


5.2.1 Legacy Wave

Table 5-2 presents the Legacy wave results for the five years it has been in existence. The savings passed

the 3% level by the second full year and have remained close to 3.5% for the last three years. It’s worth

noting the general downward trend in average household consumption during the time period. This

indicates that all households have reduced consumption, with treatment households reducing consumption

at a greater rate. It is also worth noting that although in the last two years there has been a slight increase

in savings at the household level, there was a decrease in overall program energy savings due to the effects

of sample attrition.

Table 5-2: Overall Measured Savings - Legacy Wave

Legacy Wave

Year

Average Savings (per

Household) (kWh per HH)

Average Annual

Consumption (kWh per HH)

Percent Savings Energy Savings

(kWh all Households)

2009 8 13,071 0.06% 139,836

2010 257 12,255 2.10% 4,144,843

2011 412 12,443 3.31% 6,248,830

2012 423 12,100 3.49% 6,048,460

2013 425 11,917 3.57% 5,729,750

Figure 5-1 plots the monthly savings of the Legacy wave across the five years of the program. The savings

are concentrated during the winter months with the savings level during the months of January and

February more than double that of July and August. It is also worth noting the somewhat jagged nature of

the savings during some periods. This is an artifact of the bi-monthly data.

Figure 5-1: Monthly per Household Savings Estimates (kWh)- Legacy Wave

Figure 5-2 presents the overall energy savings per month. The average per household savings is multiplied

by the count of households still receiving reports.


Figure 5-2: Monthly Overall Energy Saved (kWh) - Legacy Wave

Figure 5-3 plots the monthly percentage savings of Legacy wave households. This is the average monthly

savings divided by the average monthly consumption. This plot reiterates the importance of the non-

summer months for savings. The percentage savings plot (Figure 5-3) is similar in shape to that of the

savings plot, above, however the percentage savings are relatively constant through the non-summer

months where the savings levels continue to climb and peak during the deeper winter months. This implies

that savings scale with consumption during the winter months. This suggests savings are coming from

heating as well as lighting, assuming increased light savings during the dark winter would not be sufficient

to counteract the additional heating consumption in the denominator.

Figure 5-3: Monthly Percent Savings – Legacy Wave

5.2.2 Legacy CPW Wave

Table 5-3 presents the Legacy CPW wave results for the five years it has been in existence. The savings are

similar to the remainder of the Legacy wave through 2011 but the last two years, the savings are much


higher. The last two years, these households may have experienced additional motivation to conserve due

to the presence of the CPW program input. There is no evidence of rebate program joint savings for any of

the waves, so the interaction with the CPW program must have been of an un-tracked nature.

Table 5-3: Overall Measured Savings – Legacy CPW Wave

Legacy CPW Wave

Year

Average Savings per

Household (kWh per HH)

Average Annual

Consumption (kWh per HH)



2009 36 13,138 0.27% 195,123

2010 287 12,926 2.22% 1,559,729

2011 367 12,928 2.84% 1,938,122

2012 535 12,725 4.20% 2,651,811

2013 641 12,447 5.15% 2,997,214

We continue to see the general downward trend in average household consumption during the time period

as was seen in the remainder of the Legacy group. The substantial increase in per household savings in the

last two years has more than counteracted the effect of attrition on the overall energy savings.

The following three plots provide the monthly average per household savings, total savings and per

household percentage savings. The Legacy CPW wave is so small in size and disproportionally allocated that

the monthly results are highly variable month to month. It is difficult to develop any further conclusions on

these data.

Figure 5-4: Monthly per Household Savings Estimates (kWh) - Legacy CPW Wave


Figure 5-5: Monthly Overall Energy Savings (kWh) - Legacy CPW Wave

Figure 5-6: Monthly Percent Savings - Legacy CPW Wave

5.2.3 Expansion Wave

Table 5-4 presents the non-CPW expansion wave results for the two and a half years it has been in

existence. The savings are still in the ramp up phase seen in the first two to three years of the Legacy

groups. The savings are at a slightly lower level than the Legacy wave was at this point in its history. There

is a 10 percent difference in the second year savings, and this expansion wave was active for a longer

portion of the first year.


Table 5-4: Overall Measured Savings – Expansion Wave

Expansion Wave

Year

Average Savings per Household

(kWh per HH)

Average Annual Consumption (kWh per HH)



2011 82 11,729 0.70% 1,791,132

2012 255 11,230 2.27% 5,313,197

2013 333 11,151 2.99% 6,476,058

The following three plots provide the monthly average per household savings, total savings and per

household percentage savings. Given the shorter timeframe for the expansion group, it is more difficult to

draw conclusions from the monthly results. The winter period savings are not as dramatic for the expansion

group. In the second full winter they top out at about 40 kWh per month whereas the Legacy group reached

50 kWh per month during the winter. As should be expected, the percentage savings are also less

seasonally varied. These results indicate that the expansion group may not generate savings at the same

level as the Legacy group despite being composed of similar top-three-quartile homes.

Figure 5-7: Monthly per Household Savings Estimates (kWh) - Expansion Wave


Figure 5-8: Monthly Overall Energy Savings (kWh) - Expansion Wave

Figure 5-9: Monthly Percent Savings - Expansion Wave

5.2.4 Expansion CPW Wave

Table 5-5 presents the Expansion CPW wave results for the two and half years it has been in existence. The

savings are similar in magnitude to the non-CPW Expansion wave despite starting with 17% lower average

consumption. This accounts for the higher percentage savings. This may provide evidence that these

households also experienced additional motivation to conserve due to the presence of the CPW program

input. There is no evidence of rebate program joint savings for any of the waves, so the interaction with the

CPW program must have been of an un-tracked nature.


Table 5-5: Overall Measured Savings - Expansion CPW Wave

Expansion CPW Wave

Year

Average Savings per Household

(kWh per HH)

Average Annual Consumption (kWh per HH)



2011 56 9,714 0.57% 345,188

2012 250 9,410 2.66% 1,476,779

2013 307 9,318 3.30% 1,683,694

Figure 5-10: Monthly per Household Savings Estimate per Household (kWh) – Expansion CPW Wave

Figure 5-11: Overall Monthly Energy Saved (kWh) – Expansion CPW Wave


Figure 5-12: Monthly Percent Savings - Expansion CPW Wave

5.3 Joint Savings

In this section, we present the joint savings results. The joint savings, also referred to as uplift, represent

increased activity in other SCL energy efficiency programs caused by the HER program. Though jointly

caused by both programs, only one program can claim the savings. We identify those savings here. In this

evaluation, and in general, the double counting is avoided by netting the joint savings out of the overall

measured HER program savings. For this program, the downstream rebate savings are non-existent and the

upstream retail lighting savings are small, so the overall effect of the joint savings is small. Despite this, it is

worth remembering that they are removed from the HER program savings rather than the other program

savings only as a matter of convenience. The alternative, netting these joint savings out of the savings

claims of each affected program separately a year later when the HER program evaluation is completed,

would be more complicated.

5.4 Rebate Program Joint Savings

Figure 5-13 summarizes the average per household rebate program savings for each treatment group. The

savings start to accrue at each household when a measure is installed. Within each household, the savings

are a constant daily fraction of total claimed savings for whatever energy efficiency measures have been

installed. In combination, for the wave-level averages, the savings are a uniformly increasing stream of

accumulating savings.15 The figure is plotted on a monthly basis and only decreases for months with fewer

days.

15

The minimum measure life for any measure is 5 years. After that point, the contribution of early, short measure life measures will start to drop out

of the average per household savings calculation.


Figure 5-13: Monthly Tracked Rebate Program Joint Savings Per Household

Rebate savings are the savings that occur due to installation of energy efficiency measure following

participation in a SCL program that incentivized the measure via rebates. Joint savings from rebate

programs only occur if the savings achieved are greater for the treatment group compared to the control

group. Figure 5-14 shows the treatment and control groups’ average per household rebate program

savings for the Legacy group. In contrast to the expected finding that HER program treatment group

households would increase uptake of other SCL rebate programs, this plot shows that the average treatment

house had quite similar rebate savings during the majority of months of the program period.


Figure 5-14: Average Monthly Control and Treatment Rebate Savings for the Legacy Wave

Figure 5-15 shows the difference between the Legacy group treatment and control groups along with the

associated confidence intervals. The difference is never above a fraction of a kWh and is never statistically

significantly different from zero. Because this is based on the full treatment and control groups, this is

incorporated into the credited savings calculation as a zero. All other waves have similar rebate program

joint savings estimates, and all are counted as zero in the credited savings calculation (See the Appendix for

tables and graphs of rebate savings for the other waves). This is consistent with the experience of the other

HER program in the Seattle area, the PSE Home Energy Reports program.

Figure 5-15: Monthly Joint Savings for the Legacy Wave


Table 5-6 presents the annual joint savings estimates for all 4 waves across all years. The table includes the

joint savings estimates and their associated 95% confidence interval. If these intervals are greater than the

joint savings estimate, then the estimate is not statistically significantly different than zero. The Legacy

CPW wave is the only wave with annual joint savings of greater than two kWh. This is consistent with the

expectation that participation in the CPW program, combined with the HER reports, would result in greater

participation in other energy efficiency programs. The estimate has so much variation, however, that even

these greater magnitude joint savings are not statistically different than zero. These results indicate that

the HER program did not increase tracked savings in existing SCL energy efficiency programs.

Table 5-6: Overall Annual Joint Savings Results and Precisions

Wave Year

HER Group per Household Rebate Savings

Joint Savings Estimate per

Household, 95% confidence CI Control Treatment

Legacy

2009 0.3 0.2 0.0 (+/- 0.2)

2010 7.5 8.3 0.8 (+/- 2.2)

2011 18.5 17.2 -1.3 (+/- 3.4)

2012 28.3 28.2 -0.1 (+/- 4.3)

2013 51.0 52.3 1.3 (+/- 6.3)

Legacy CPW

2009 1.1 1.0 -0.1 (+/- 0.8)

2010 9.6 11.3 1.8 (+/- 6.6)

2011 10.5 16.6 6.1 (+/- 8.3)

2012 20.3 21.7 1.4 (+/- 18.8)

2013 36.9 42.1 5.2 (+/- 23.3)

Expansion

2011 1.2 1.6 0.3 (+/- 0.6)

2012 12.6 11.9 -0.8 (+/- 3.5)

2013 34.1 35.2 1.2 (+/- 5.7)

Expansion CPW

2011 1.2 0.5 -0.7 (+/- 1.1)

2012 9.7 5.3 -4.4 (+/- 6.3)

2013 26.0 24.6 -1.4 (+/- 9.4)

5.5 Upstream Retail Lighting Program Joint Savings

Section 4.2.2, Upstream Retail Lighting Joint Savings, provides the methods and estimates for the upstream

program joint savings. These estimates are based on estimates produced by PSE for the evaluation of their

HER program. The results from those evaluations are extremely good proxies for the SCL upstream joint

savings.


6 CONCLUSIONS

The Seattle SCL HER program has succeeded at reducing household consumption by promoting behavioral

change and other conservation actions. The RCT experimental design, in combination with the large sample

sizes, provides high precision, un-biased estimates of program savings. These savings are the core of an

evaluation that is as robust as any energy program evaluation despite the small magnitude of the per

household savings.

This evaluation estimated the annual measured savings for all years of all four HER program waves. Rebate

program joint savings were estimated and found to be not statistically different than zero. Upstream joint

savings estimates from Puget Sound Energy’s HER program evaluation were used to net out potential double

counting with the Twist and Save retail lighting program. We calculated credited savings for 2012 and 2013

to represent the HER program savings net of any potential double counting. This estimate of electricity

savings reflects the unique effect of the Seattle SCL HER program.

6.1 Recommendations

SCL should consider doing further analysis on certain aspects of the HER program. Survey analysis could

provide further information on customer satisfaction regarding the reports as well as a better understanding

of what kind of actions proceed from the reports.

In particular, it would be interesting to further explore the combined effect of the HER program and the CPW

program. In combination, the two programs appear to have generated additional savings above and beyond

the savings that could be expected from either program individually.

Survey analysis could also directly measure the uptake of the Twist and Save retail lighting program. The

recent availability of LED bulbs and fixtures has re-energized these retail programs. These bulbs give the

HER reports a tangible target with which to generate savings. It would be useful to have a SCL-specific

measurement of the additional uptake of this program due to the HER reports.


7 APPENDIX

7.1 Tabular Measured Savings

Table 7-1: Monthly Measured Savings - Legacy Wave

Month Impact Estimate

(per Household)

Energy Savings

(kWh)

10/1/2009 -0.96 (16,238)

11/1/2009 19.28 325,385

12/1/2009 6.23 104,626

1/1/2010 15.13 252,719

2/1/2010 19.75 328,450

3/1/2010 20.77 343,763

4/1/2010 12.07 198,405

5/1/2010 15.41 251,545

6/1/2010 2.13 34,480

7/1/2010 14.41 231,974

8/1/2010 4.39 70,225

9/1/2010 19.79 314,680

10/1/2010 13.83 218,595

11/1/2010 48.00 754,565

12/1/2010 46.51 727,285

1/1/2011 42.48 661,289

2/1/2011 35.42 549,762

3/1/2011 40.99 633,040

4/1/2011 20.84 320,381

5/1/2011 20.65 315,517

6/1/2011 11.09 168,548

7/1/2011 17.72 267,720

8/1/2011 13.48 202,081

9/1/2011 22.81 340,055

10/1/2011 23.57 349,578

11/1/2011 59.67 881,278

12/1/2011 64.79 952,322

1/1/2012 60.14 880,460

2/1/2012 43.18 629,892

3/1/2012 39.38 572,007

4/1/2012 26.21 379,038

5/1/2012 22.95 330,213

6/1/2012 18.35 262,486


Month Impact Estimate

(per Household)

Energy Savings

(kWh)

7/1/2012 30.46 432,733

8/1/2012 13.95 196,932

9/1/2012 23.51 329,762

10/1/2012 22.64 316,093

11/1/2012 57.40 797,933

12/1/2012 57.52 796,633

1/1/2013 51.24 707,544

2/1/2013 49.05 674,449

3/1/2013 53.53 731,910

4/1/2013 38.38 521,881

5/1/2013 25.09 339,780

6/1/2013 12.95 174,327

7/1/2013 26.06 348,130

8/1/2013 5.67 75,162

9/1/2013 28.96 382,256

10/1/2013 30.52 400,968

11/1/2013 63.78 835,681

12/1/2013 46.01 601,941


Table 7-2: Monthly Measured Savings - CPW Expansion

Month

Impact Estimate

(per Household)

Energy Savings

(kWh)

6/1/2011 5.76 36,992

7/1/2011 1.41 8,999

8/1/2011 -1.61 (10,156)

9/1/2011 6.64 41,524

10/1/2011 9.56 59,304

11/1/2011 3.44 21,213

12/1/2011 30.52 187,312

1/1/2012 17.48 106,858

2/1/2012 35.24 214,238

3/1/2012 15.35 92,763

4/1/2012 35.90 215,765

5/1/2012 21.41 127,734

6/1/2012 16.49 97,765

7/1/2012 15.57 91,585

8/1/2012 7.02 41,030

9/1/2012 13.79 79,944

10/1/2012 16.30 93,601

11/1/2012 16.34 93,123

12/1/2012 39.32 222,373

1/1/2013 38.71 217,249

2/1/2013 48.67 271,845

3/1/2013 38.52 214,377

4/1/2013 37.99 210,033

5/1/2013 24.28 133,758

6/1/2013 22.38 122,329

7/1/2013 8.89 48,167

8/1/2013 6.60 35,467

9/1/2013 -1.50 (8,012)

10/1/2013 19.33 102,588

11/1/2013 23.43 123,871

12/1/2013 40.19 212,024


Table 7-3: Monthly Measured Savings – Non-CPW Expansion

Month

Impact Estimate

(per Household)

Energy Savings

(kWh)

6/1/2011 4.39 99,408

7/1/2011 -0.59 (13,262)

8/1/2011 7.28 162,145

9/1/2011 5.28 116,855

10/1/2011 20.91 459,274

11/1/2011 14.24 310,954

12/1/2011 30.18 655,758

1/1/2012 15.10 326,665

2/1/2012 33.51 721,566

3/1/2012 21.08 451,490

4/1/2012 25.15 535,260

5/1/2012 19.06 402,366

6/1/2012 22.75 477,213

7/1/2012 12.24 254,722

8/1/2012 19.13 394,516

9/1/2012 8.29 169,851

10/1/2012 21.07 428,483

11/1/2012 20.59 416,456

12/1/2012 36.56 734,609

1/1/2013 41.41 827,508

2/1/2013 36.89 733,944

3/1/2013 39.84 788,140

4/1/2013 33.57 660,676

5/1/2013 17.59 343,761

6/1/2013 29.04 563,329

7/1/2013 14.79 284,884

8/1/2013 20.18 385,997

9/1/2013 14.93 283,844

10/1/2013 26.26 496,233

11/1/2013 20.94 394,009

12/1/2013 38.00 713,733


7.2 Joint Savings Plots

Figure 7-1: Average Monthly Control and Treatment Rebate Savings – Expansion CPW

Figure 7-2: Monthly Joint Savings – Expansion CPW

0

0.5

1

1.5

2

2.5

3

Control Treatment

-1.5

-1

-0.5

0

0.5

1

1.5

Lower Limit Joint Savings Upper Limit


Figure 7-3: Average Monthly Control and Treatment Rebate Savings – Legacy CPW

Figure 7-4: Monthly Joint Savings – Legacy CPW

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

51

0/1

/20

09

12

/1/2

00

9

2/1

/20

10

4/1

/20

10

6/1

/20

10

8/1

/20

10

10

/1/2

01

0

12

/1/2

01

0

2/1

/20

11

4/1

/20

11

6/1

/20

11

8/1

/20

11

10

/1/2

01

1

12

/1/2

01

1

2/1

/20

12

4/1

/20

12

6/1

/20

12

8/1

/20

12

10

/1/2

01

2

12

/1/2

01

2

2/1

/20

13

4/1

/20

13

6/1

/20

13

8/1

/20

13

10

/1/2

01

3

12

/1/2

01

3

kWh

Control Treatment

-4

-3

-2

-1

0

1

2

3

4

10

/1/2

00

9

12

/1/2

00

9

2/1

/20

10

4/1

/20

10

6/1

/20

10

8/1

/20

10

10

/1/2

01

0

12

/1/2

01

0

2/1

/20

11

4/1

/20

11

6/1

/20

11

8/1

/20

11

10

/1/2

01

1

12

/1/2

01

1

2/1

/20

12

4/1

/20

12

6/1

/20

12

8/1

/20

12

10

/1/2

01

2

12

/1/2

01

2

2/1

/20

13

4/1

/20

13

6/1

/20

13

8/1

/20

13

10

/1/2

01

3

12

/1/2

01

3

kWh



Figure 7-5: Average Monthly Control and Treatment Rebate Savings - Expansion

Figure 7-6: Monthly Joint Savings – Expansion

0

0.5

1

1.5

2

2.5

3

3.5

4

kWh

Control Treatment

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1



7.3 Monthly Attrition

Figure 7-7: Monthly Move Outs - Legacy Wave


Figure 7-8: Monthly Move Outs - Legacy CPW Wave


Figure 7-9: Monthly Move Outs - Expansion Wave


Figure 7-10: Monthly Move Outs - Expansion CPW Wave

ABOUT DNV GL Driven by our purpose of safeguarding life, property and the environment, DNV GL enables organizations to advance the safety and sustainability of their business. We provide classification and technical assurance

along with software and independent expert advisory services to the maritime, oil and gas, and energy industries. We also provide certification services to customers across a wide range of industries. Operating in more than 100 countries, our 16,000 professionals are dedicated to helping our customers make the world safer, smarter and greener.

Home Electricity Report Program, January 2012 …...HOME ELECTRICITY REPORT PROGRAM JANUARY 2012...

Documents

Transcript of Home Electricity Report Program, January 2012 …...HOME ELECTRICITY REPORT PROGRAM JANUARY 2012...