Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Evaluation and Program Planning, Vol. 6, pp. 121-129, 1983

Printed in the USA. All rights reserved.

0149-7189/83 $3.00 + .OO

Copyright 0 1984 Pergamon Press Ltd

EVALUATIONS OF UTILITY RESIDENTIAL CONSERVATION PROGRAMS IN THE PACIFIC NORTHWEST

A Critical Review

LINDA BERRY

and

KIM-ELAINE JOHNSON

Energy Division Oak Ridge National Laboratory

Oak Ridge, Tennessee 37830

ABSTRACT

Pacific North west utilities have sponsored the nation’s earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.

INTRODUCTION

Utility energy conservation programs represent an important and rapidly growing element of electric utility activities. For example, the Tennessee Valley Authority provided home energy audits to more than 490 thou- sand (about 20%) of their residential customers between August 1977 and September 1981 (Tennessee Valley Authority, Note 1). California electric and gas utilities spent $157 million in 1981 on their customer conservation programs, up from $91 million in 1980 (Amaroli, Note 2). The federal Residential Conserva-

tion Service (RCS), mandated by the National Energy Conservation Policy Act (1978), also is significantly increasing the amount of utility conservation activity and the demand for evaluation of these activities (Soderstrom, Berry, & Hirst, 1981). As utility conservation activities increase, utilities are devoting more attention to determinations of energy-saving program impacts and their cost effectiveness (to participating customers, to nonparticipating customers, and to utility stockholders).

UTILITY EVALUATIONS IN THE PACIFIC NORTHWEST

Pacific Northwest utilities have sponsored the nation’s earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility program evaluations.

The authors recently completed a metaevaluation of Northwest utility program evaluations. This metaevaluation was conducted as part of an effort to develop a plan (Hirst, Berry, Bronfman, Johnson, Tepel, &

Trimble, 1982) for a Bonneville Power Administration (BPA) residential conservation program evaluation (see Soderstrom et al., 1981, for a general discussion of the value of metaevaluation for evaluation planning purposes). Findings from this BPA-sponsored metaevaluation also are applicable to RCS evaluation planning and supplement an earlier review of home energy audit program evaluations (Soderstrom et al., 1981).

The Bonneville Power Administration (BPA) serves

Research sponsored by the Office of Conservation and Direct-Application Renewable Resources, Bonneville Power Administration, U.S.

Department of Energy under contract W-7405eng-26 with Union Carbide Corporation.

Requests for reprints should be sent to Linda Berry, Energy Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830.

121

122 LINDA BERRY and KIM-ELAINE JOHNSON

the Northwest region’s electric load through wholesale arrangements with 124 utilities, six Federal agencies and 17 large industries. With impetus from the 1980 Pacific Northwest Electric Power Planning and Con- servation Act (P.L. 96-501) BPA is funding residential conservation programs that are implemented by participating utilities. Some elements of program design are required by BPA; others are chosen by the participating utilities.

home; and view insulation as being more effective than conservation practices. Additionally, nonparticipants are more frequently over the age of 65 and are more frequently “do it yourselfers.”

As a first step in developing an evaluation plan for these BPA residential conservation programs, the authors contacted most BPA-area utilities in Septem- ber-November 1981 to gather information about their residential conservation programs, past evaluations, and evaluation plans. (A complete summary of findings is available in Appendix B of Hirst et al., 1982.)

A number of utilities in the BPA region supplied us with formal conservation program evaluation documents including: Pacific Power and Light (PP&L) (Pacific Power and Light, Notes 3 & 4); Portland General Electric (PGE) (Burnett, Notes 5, 6, 7, & 8); Puget Sound Power and Light (Puget) (Croft, Note 9); and Seattle City Light (SCL) (Bradley & Shaffer, Note 10; Olsen & Cluett, Note 11; Weiss & Newcomb, Note 12). These evaluation documents are reviewed in detail in this paper. While four area utilities have produced formal evaluation documents (PP&L, PGE, Puget, and SCL), over a dozen BPA-area utilities have conducted in-house evaluation activities and have further evaluations planned or in progress. The evaluation activities of these utilities are not discussed in this paper because they raise no issues not included in the formal evaluation reports.

Three major issues are addressed in one or more of the BPA-area utility program evaluations we received: (1) Who participates in the programs and why? (2) How much energy is saved as a result of the programs? (3) How cost effective are the programs? Our discussion of the utility evaluation efforts is organized into sections by these three issue areas. Some evaluations covered several issues and some only one. If an evaluation is relevant to several issues the pertinent part will be discussed for each issue. After a review and critique of the evaluation findings dealing with each issue, some general conclusions and recommendations for future conservation program evaluation efforts are presented.

These typical profiles of participants and nonparticipants were confirmed by the BPA-area studies we reviewed. The PGE study (Burnett, Note 6) found, for example, that the typical participant was a higher- than-average consumer of electricity, was younger and more educated than average, had a moderately high income and a larger-than-average family. The nonparticipants were more likely to be renters, older/retired persons, and younger families with a two-person income and no one home during the day.

The PGE study also examined trends in the type of person who weatherizes through PGE’s zero interest loan program. Two groups of participants were surveyed: those weatherizing early in the program and those weatherizing later in the program. No differences were found between the two groups in average dwelling size or average consumption. The later participants were somewhat younger and had more educa- tion and slightly higher incomes, but these differences might well be due to sampling error.

The PGE study was based on data derived from three main sources: (1) personal interviews with 758 single-family household PGE customers (508 participants and 250 nonparticipants) conducted during the period of January-March 1981, (2) a 1977 survey of dwelling characteristics where PGE inspected the insulation of the structure (for nonaudited customers only), and (3) PGE weatherization audits.

The Puget Power survey (GMA Research Corpora- tion, Note 13) was designed to identify barriers which inhibit requests for weatherization audits under Puget’s program. The survey was conducted with ten specific objectives in mind. The objectives differed, in most cases, for respondents who had requested audits from those who had not. These objectives were:

For all Respondents

1. To determine opinions and attitudes toward energy issues.

2. To collect demographic and structural information.

Who Participates and Why For Non-audit Respondents A variety of characteristics have been found to distin- guish utility conservation program participants from nonparticipants (Berry, Soderstrom, Hirst, Newman, & Weaver, 1981). Participants are more likely to: live in older, larger homes; have higher electric bills; be better educated; have higher incomes; have young children at

3. To determine the level of weatherization features present in homes. (Weatherization features include attic, floor and wall insulation, storm doors and windows, caulking and weatherstrip- ping, and water heater wraps.)

Evaluations of Utility Residential Conservation Programs 123

4. To determine which conservation measures

have been undertaken and when. (Conservation measures may include the installation of weath-

erization features such as those listed above and also actions such as reducing thermostat set- tings, using appliances more efficiently, or installing clock thermostats or shower flow restrictors.)

5. To determine the future likelihood of installing specific conservation devices.

6. To determine the level of awareness of Puget’s program.

7. To determine future likelihood of program participation.

For Audit Respondents

8. To obtain feedback on satisfaction with the program.

9. To determine which improvements the auditor suggested and the likelihood of implementing those suggestions.

10. To determine factors which influence home- owners to apply or not apply for zero-interest loans.

Toward these ends, two methods were employed. In

order to determine consumer attitudes, opinions, and concerns, focus group meetings were conducted during August 1981. The issues identified as important in the focus groups provided a qualitative foundation for the design of a quantitative study based on a telephone survey instrument. The quantitative study is made up of means analyses and frequency tabulations of the telephone survey which was conducted in September and October 1981.

The sample for the telephone survey consisted of 500 subjects who owned their own homes, heated with electricity, and did not work for an advertising/marketing research firm or an electric utility. Thus, only customers who were eligible for Puget Power weatherization were included in the study.

Several important barriers and relationships between characteristics were identified. It was found, for example, that the main reason households had not requested Weatherization Checkups was a perception that their homes were adequately energy efficient and that people who had higher bills and owned larger homes were more likely to request Checkups. It also was found that participation would likely be increased

by offering audits on weekends, offering discounts on weatherization work in lieu of zero-interest loans, and providing a check-off box on bills to request an audit.

The PGE and Puget studies both define the same profile of audit program participants. In fact, all studies of participant/nonparticipant characteristics seem to define the same basic profiles (Berry et al., 1981). In short, numerous surveys have provided clear and consistent results on the question of who does and does not participate in audit programs.

Even though findings about participant/nonparticipant characteristics have been consistent, this does not mean that further research on this issue is unnecessary. It is likely that audit programs to date have “skimmed the cream” from the market for conservation retrofit. That is, the profile of audit participants is the same as the typical profile of early adopters of most innova- tions (Rogers & Shoemaker, 1979). This market seg- ment is the easiest to motivate and reach. Penetration into other market segments will be more difficult and may require different strategies. Research on how to motivate groups that have not responded in the past such as renter-occupied, older-retired, and younger- two-income households is needed to penetrate these market segments.

As new marketing strategies are implemented, re-

peated measurements of participant characteristics will be needed to monitor the effectiveness of the strategies. Understanding what types of households are being reached at various stages of program development also is important as an aid to load forecasting efforts. As new market segments are reached, conservation program impacts are likely to vary. It may be that households with the highest potential for savings participate early in the program and that, therefore, program impact will diminish over time. Similarly, it may be that the number of participants will reach a peak level at about the third or fourth year of program operation and then decline as the market saturates. Understand- ing how participation rates among various groups change over time and how the likely savings achieved by early/late participants will differ can make forecasts of program impacts on loads more accurate.

While the issue of who participates and why will need continued attention, the methods available to study it seem to be adequate. As both the PGE and Puget studies illustrate, research on differences in participant/nonparticipant characteristics does not require complicated research designs. Comparisons of simple random samples drawn from both groups are appropriate.

Energy Savings Seven Pacific Northwest utility studies that estimated energy savings due to conservation programs (Pacific Power and Light, Notes 3 & 4; Burnett, Note 8; Croft,


Note 9; Olsen & Cluett, Note 11; Bradley & Shaffer, Note 10; Weiss & Newcomb, Note 12) are reviewed in this section. In these studies the estimates of average annual kWh savings per participating household vary from 1534 kWh (Weiss & Newcomb, Note 12) to 7114 kWh (Croft, Note 9), (see Table I).

Three of the studies attributed all weather-adjusted changes in participant consumption to the program’s influence (Pacific Power Light, Notes 3 and 4; Croft, Note 9). In these studies either no comparison group was used (Pacific Power and Light, Note 3; Croft, Note 9) or no adjustments to energy savings estimates were made on the basis of observed changes in comparison group consumption (Pacific Power and Light, Note 4). The first PP&L study (Pacific Power and Light, Note 3) for example, used no comparison group to estimate the amount of savings due to factors such as higher prices, changing attitudes toward energy use (a conservation ethic), or private weatherization efforts. AH savings calculated by the means analysis were assumed to be due to the program. Similarly, in the Puget study (Croft, Note 9) of savings no control

group was used to determine trends in energy use for the utility service area. With temperature adjusted annual savings at an average of 7114 kWh per participating household, it seems likely that something was changing besides program-induced additions of conservation retrofits. No information on wood stove additions, on changes in price, income, and numbers of household members, or on other factors influencing consumption was used to adjust savings estimates, however.

It is difficult to have much confidence in estimates of program-induced energy savings that do not make use of a comparison group. Measurement of the energy savings which are attributable to a particular program (as distinct from other forces driving conservation) is a complex undertaking. A household’s demand for energy has many determinants including fuel prices, weather, appliance stock holdings and usage patterns, weatherization features of the dwelling unit, and behavioral characteristics of the occupants. Ail of the determinants of demand interact with the influence of the utility program intervention to produce changes in

TABLE 1

COMPARISON OF ENERGY-SAVINGS ESTIMATES DUE TO PACIFIC NORTHWEST UTILITY PROGRAMS

Evaluation Comparison Group Average Annual kWh Saved per

Adjustment Factors Participating Household

Puget Residential Weatherization

Program (Croft, Note 9)

None Weather 7114 kWh

PP&L Residential Conservation Program (Pacific Power and

Light, Note 4)

Consumption of a sample Weather 1737 kWh HEAa only of nonparticjpants was 2218 kWh HEA + WHWb examined but not used to 4105 kWh HEA + Weatherization adjust savings estimates 4586 kWh HEA + WHW +

Weatherization

PP&L Zero Interest Weatherization Program (Pacific Power and Light,

Note 3)

None Weat her 4621 kWh

PGE Weatherization Program

(Burnett, Note 8)

SCL Neighborhood Program (Olsen 8 Cluett, Note 11)

Random sample of Weather 3929 kWh nonparticipants Vacancy ratesC

Nonparticipants who had None 2344 kWh no type of program contactd

SCL Residential Insulation

Program (Bradley & Shaffer,

Note 10)

Nonparticipants with a

distribution weighted by

prior consumption

Prior consumption 1900 kWh

SCL Home Energy Check Program Later participants None 1.534 kWh (Weiss & Newcomb, Note 12)

aHome Energy Audit. bWater Heater Wrap. cln this study, Burnett found that participant homes were more likely to have been vacant before an audit request was made. To remove the resulting downward bias on estimates of program impadts he eliminated all households that had been vacant from- the sample. This procedure probably accounts, in part, for the fact that the PGE estimate of savings is higher than the SCL

Neighborhood Program estimate. dNot even discussions with neighbors.


energy consumption. Estimating the amount of change in consumption due to the program requires some effort to hold all else equal. Use of a comparison group is an important means of disentangling program impact from numerous confounding factors.

The PGE study of impact (Burnett, Note 8) used a comparison group of randomly selected nonparticipants. In this study, it was recognized that participant and nonparticipant dwellings may-in their sensitivity to weather-differ and therefore, separate weather- adjustment models were developed for each group. An adjustment also was made for higher participant pre- program household vacancy rates by eliminating from the sample all households that had been vacant. The removal of vacant households probably accounts, in part, for the fact the PGE estimates of savings are somewhat higher than the SCL estimates (see foolnote to Table 1).

The three SCL evaluations (Olsen & Cluett, Note 1 I; Bradley & Shaffer, Note 10; Weiss & Newcomb, Note 12) each used the consumption patterns of some sort of comparison group to adjust estimates of program- induced energy savings. In the evaluation of the Seattle City Light Neighborhood Energy Conservation program, the comparison group consisted of households that did not take part in any program activity (a variety of activities including block workshops, neighborhood campaigns, and audits were offered) and that did not discuss conservation with neighbors. The lack of a sys- tematic selection procedure makes this group a ques- tionable source of comparison. The XL households that had no contact, however indirect, with the program are probably even more different from participants than the typical nonparticipant would be. As a result, differences in the composition of the participant and comparison groups could account for much of the observed difference in their levels of consumption. In other words, the influence of the program is only one possible explanation for the greater reduction in usage among the participant group (6.2%) than among the nonexposed comparison group (1.8%).

In the evaluation of the SCL Residential Insulation Program (RIP) the comparison group was selected somewhat more systematically. The comparison group of nonparticipants consisted of 55 1 electric-heat customers who were randomly_ selected as part of a 5,000-household sample for the 1979 Attitude and Awareness Survey. The participant group consisted of all 278 households insulated through RIP as of September 1980. Because average consumption of the participant group was (in a chi-square test) significantly higher in pre-program heating seasons than that of the nonparticipant group, the control group was weighted to approximate the RIP distribution of customers according to consumption. A paired i-test was used to determine the significance of differences in

the two groups’ consumption changes between the 1978-1979 and 1979-1980 heating seasons. Since the changes in consumption were significant for both groups, the consumption changes occurring in the control group were subtracted from those of the RIP group to produce the net kWh savings attributable to RIP. A distinction was made in the analysis between full and partial electric heat RIP participants because the 34 partial electric heat participants had an average increase in consumption of 71% (largely because of switching to a higher proportion of electric heat). In contrast, the full electric heat RIP participants had a net consumption decrease of 7%. The net change in consumption was obtained by subtracting the change occurring by consumption range in the control group from the change by consumption range in the RIP group.

It is difficult to assess the potential problems with the weighting and statistical procedures used in this analysis of RIP because they were not described in any detail. There was no discussion, for example, of how the weighting procedure might bias the results. There also was no information on the sampling frame used for the 1979 survey. The application of a weighting procedure to a sample for which probabilities of selection from each stratum are not known may introduce unaccountable biases.

The effort to weight on prior consumption levels represents, however, a significant advance over methods of estimating savings that give no considera- tion to any confounding factors except weather. It also is an improvement over comparison group selection procedures that do not examine differences between the participant and comparison groups on potentially confounding factors.

In the evaluation of the SCL Home Energy Check (HEC) program (Weiss & Newcomb, Note 12), the comparison group consisted of households volunteer- ing to participate in the program at a later time. In this design, which is shown in Table 2, households requesting audits during the first year of the study period form the participant group and households requesting audits in the second year form the comparison group. Comparisons of consumption for the two groups at time T3 are the measure of program impact.

TABLE 2 MULTIPLE BASELINE DESIGN

T, T* T3 T4

01 02 x 03 (group 1) “Participants”

0, 03 x 04 (group 2) “Comparison group”

T,., = four time periods each one year apart. 0,_4 = yearly observations of energy consumption. X = audit received.


This method of selecting a comparison group pro- duces a group more similar to participants than any discussed before. In fact, the comparison group members are essentially “participants” who have not yet been exposed to the program. They are likely to have the same general distribution of characteristics as all other participants. This similarity between the two groups makes it possible to more effectively separate program effects from other factors influencing consumption.

Of course, it is possible that households requesting audits at different times have different characteristics. Later participants might, for example, have less interest in conservation or be less sensitive to price increases. In the HEC study (Weiss & Newcomb, Note 12) this possibility was considered and checked by comparing preprogram consumption levels for the two groups. Since the two groups had nearly identical pre-program consumption levels, it was concluded that they were not significantly different from each other.

it is interesting to note that in the three SCL studies the more closely the comparison group resembled the participant group the less savings were attributed to a program. When the comparison group consisted of those households which had no contact with an SCL program activity, not even a conversation about it with neighbors, the savings were estimated as 2344 kWh (Olsen & Cluett, Note 111, (see Table I). When the comparison group was matched to the participant group on prior consumption, savings were estimated as 1900 kWh (Bradley & Shaffer, Note 10). When the comparison group consisted of households which requested an audit at a later date than the households defined as early part.icipants, savings were estimated as 1534 kWh (Weiss & Newcomb, Note 12).

The three SCL evaluations were, however, designed to measure the impacts of three different SCL programs. Thus, the difference in findings could be due to different program features as well as to different definitions of comparison groups. Nevertheless, the pattern of findings does suggest that the early/late participant approach to defining a comparison group reduces the influence of confounding factors more than the other methods do.

Since factors that determine energy consumption are distributed differently between participant/nonparticipant groups, estimates of program-induced energy savings that do not consider these differences in group composition are likely to be misleading. Because the early/late participant design defines a comparison group more like the participants than any other study design, it seems likely that it produced the most valid estimates of program impact.

Cost Effectiveness Two SCL evaluations (Bradley & Shaffer, Note IO; Weiss & Newcomb, Note 12) considered the issue of

cost effectiveness. In both studies the cost effectiveness af the programs (RIP and HEC) was assessed with Net Present Value (NPV) analysis. Net Present Value analysis (NPVf is the classical technique used in economics for investment decisionmaking. Generally it takes account of the “time preference for money” in decisions which involve giving up present consumption (investing} for future benefits. “Time preference for money” means that an individual or institution will prefer to have (for example) one dollar now to spend rather than the same amount 1 year from now. If an investment is to be undertaken it must pay back to an individual or institution an amount greater than was originally invested. NPV analysis takes al1 the benefits and costs of an investment and calculates their worth in a particular year. This allows the comparison of costs and benefits.

Application of NPV analysis to utility-sponsored conservation programs, though useful, raises a number of thorny issues. At the most general ievel, the issue of what costs and benefits to include in the analysis must be resolved. NPV calculations often ig- nore some important benefits of conservation such as its environmental impacts in comparison to supply alternatives. In the studies reviewed here, for example, only the monetary value of deferred power plant con- struction is induded in estimates of regional benefit. The value of environmental effects is nat considered. Choices about what costs and benefits to include in NPV calculations are essentially value choices which will always be open to debate.

A second important source of difficulty in NPV analysis is the sensitivity of results to the assumptions used. Small differences in the assumed discount rate or the assumed marginal cost of electricity, for example, can have large effects on the resulting estimates of cost effectiveness. This sensitivity is especially problematic because the choice of assumptions is itself highly judg- mental and uncertain. Similarly, the estimate used of the savings due to the program will strongly influence the results of a cost effectiveness analysis. But, this input, like many others, is itself difficult to obtain with a high degree of accuracy.

Since the correctness of NPV estimates depend on the validity of program savings estimates and on a large number of assumptions that may be difficult to justify, cost-effectiveness results must be interpreted cautiously. Assessment of a program’s worth is ex- tremely complicated. One needs not only reliable information on program costs and benefits including energy savings, but also reliable information on the marginal costs of power production, transmission, and distribution-as a function of time (i.e.> of seasonal and daily variations) (Hirst, Bronfman, Go&s, Trimble, & Lerman, 1983).

The NPV analyses done by SCL were aimed at determining whether their pmf efforts in investing in


their RIP and HEC programs were worthwhile, that is, if the benefits derived from the program were greater than the costs. Both NPV analyses done by XL included the following steps:

1. Costs and benefits (and when they occur) were determined for the 2%year lifetime of the insulation. The nonrecoverable expenditures include administrative costs and loan discounts. The principal of the loans will be repaid over time. Benefits were calculated as average annual capacity savings (kW) and were classified as high demand period (peak and intermediate), off- peak winter period and summer period to reflect the cost of providing the electricity.

2. Monetary values were assigned to the costs and benefits.

3. Past costs and benefits were put in 1980 dollars using the Consumer Price Index to remove the effect of inflation.

4. Real future costs and benefits were discounted by 3% per year.

5. The present value of costs and of benefits were compared to determine the NPV of RIP.

For the RIP program the NPV was calculated from three perspectives: the regional, the utility, and the participant. For the HEC program only the regional and utility perspectives were considered. There are differences in the costs and benefits associated with each of the three perspectives. The benefits to the region are the average annual capacity savings translated into dollars over the lifetime of the weatherization actions taken. For the utility the benefits are defined the same way but are smaller because the marginal cost of electricity to Seattle City Light is less than the marginal cost of electricity to the region as a whole. The benefit to the participant consists of monetary savings on electric bills based on projections of future average residential rates.

Because of the differences in definitions of costs and benefits reviewed above, RIP had a different NPV for

each of the three perspectives. All of the perspectives, however, had a positive NPV per participant (for fully electric households). The NPV of RIP for the region was found to be $1,308 per participant and for the utility $171 per participant. For the average participant RIP had a NPV of $356. The NPV of HEC for electrically-heated homes for the region was found to be $475 per participant and for the utility $138 per participant.

Overall, these SCL applications of NPV analysis are commendable. Their use of marginal fuel costs is especially praiseworthy. Their choice of methodology and its application result in some of the better cost- benefit analyses of weatherization programs available.

In spite of the overall quality of the analysis, however, there are a few criticisms that can be made. First, the assumptions chosen seem to be somewhat favorable to the programs. For example, the real discount rate and the real fuel escalation rate used were 3% and 2.46%, respectively. These effects net out in the analysis so that future fuel savings are, in effect, discounted at about 0.5%. Conservation investments are risky due to their long term nature, low liquidity, and the variability of fuel prices. Therefore, 15% real discount rates are not unusual. The Office of Manage- ment and Budget requires projects to be evaluated at a 10% real rate. The real fuel escalation rate, on the other hand is higher than many analysts assume. Often electricity is assumed to have a 0.0% real escalation rate; but given the problems with nuclear plant con- struction in the Northwest, the rate of 2.46% may not be unreasonable.

A second point concerns the applicability of the studies’ results to further extensions of the programs. Because of the way the programs were offered to participants, it is likely that the first group to respond is most likely to gain from the program, i.e., it was made up of households with high electrical usage and low insulation. (Hirst et al., 1983, show that the BPA pilot program participants had higher electrical usage and lower levels of insulation than nonparticipants.) This is commonly referred to as self-selection bias. The calculated NPV might not be as high with succeeding participants. A lower bound could be roughly estimated by weighting the study group to look like the control group.

CONCLUSIONS

Each of the utility evaluation studies reviewed in this and PGE studies, for example, focused on attitude paper makes a contribution to the problems of analyz- measurement and marketing issues. Energy savings ing the operations of and/or estimating the impacts of were estimated by seven of the utility studies, but only residential conservation programs. Even though all of the XL study of the HEC program used an adequate the evaluations use the same types of data (customer comparison group. Cost effectiveness issues were ad- surveys, audit data, and billing records) their methods dressed in only two studies: SCL’s RIP and HEC of analysis and findings vary considerably. The em- evaluations. phasis given to specific issues also differs. The Puget The Puget and PGE efforts to compare the social,


economic, and demographic characteristics of participants/nonparticipants and of early/late participants were quite successfuf. and a consistent profile of the typical participant emerged. The question of how participants/nonparticipants differ is, of course, much easier to address than the energy savings or cost- effectiveness issues, That is, only simple descriptive information is required and the measurement and control of numerous confounding factors is not necessary.

As a whole, the evaluations suggest a need for further methodological development in the area of estimating energy savings attributable to programs. Most of the studies adjusted only for the confounding effects of weather. It is clear that other potential sources of bias must be examined too. disentangling the confounding effects of factors such as weather, fuel prices, and differences between participant/nonparticipant groups is as difficult as it is important in unambiguously estimating the energy savings for a particular program, net of ail other effects.

Use of the early/fate participant group design in the XL HEC study probably produced the most reason- able estimate of program energy savings impacts. This method is recommended for future evaluation efforts. An even better approach would be some form of con- trolled experiment such as a design that randomly assigned participants to a waiting list (see Soderstrom et al,, 1981 and Berry, 1983, for discussions of some experimental design options). While a good evaluation design is essential to obtaining valid estimates of savings due to the program, it should be supplemented with some form of multiple regression analysis. Multi- pte regression is an essential supplement because it allows for analysis of the relationships among factors that influence consumption. The early/late participant

design provides estimates of the size of conservation program impacts. But, unlike regression analysis, it does not allow one to weight predictor variables or to adjust for specific confounding factors.

Net Present Value analysis is a suitable technique for dealing with cost effectiveness issues. The validity of the results depends, however, on a whole series of assumptions (about discount rates, future fuel prices, inflation rates, etc.) that sometimes may be difficult to justify. In addition, the estimates of how much energy savings are attributable to the program must be accurate if the Net Present Value (NPV) of the program is to be calculated correctly. Capacity planning models that can determine the vafue of given amounts of conservation program savings relative to supply alternatives also provide essential input to NPV analysis. These models generally involve computer simulations that plan the generation requirements and costs of a range of forecasted demand levels. With these models the value of conservation program savings are quan- tified with respect to variations in future load growth, expansion plans and projected finances and rates. Thus, the correctness of NPV estimates depends on many assumptions and on the validity of the results obtained from other analytical efforts. Careful attention to the quality of these inputs to NPV analyses is necessary if one is to have confidence in the results.

Overall, the Pacific Northwest evaluations we reviewed present a promising picture of the state of the art in evaluation of utility residential conservation programs. There is a good deal of consensus on the major research issues, methods, and probiems. Some very good studies have been complted. Further refinements will, of course, be needed; but the studies reviewed here provide a basis for continued progress.

REFERENCE NOTES

i.

2.

3.

4.

5.

TENNESSEE VALLEY AUTHUKKTY, DIVISION OF

ENERGY CONSERVATION AND RATES, OFFICE QF

POWER. Program summary. October 1981.

AMAROLI, C. 1980 Energy savings and peak demand red~c- tions, and 19%-f981 expenses for energy conservation pro- gmms. Memorandum to members of the California Public

Utility Commission, October 29, 1981.

PACIFIC POWER AND LIGHT COMPANY. Analysis of oc- tual savings from Oregon zero interest weatkerization program at Pacifc Power and Light Company. 1981. (Draft)

PACIFIC POWER AND LIGHT COMPANY. ~es~dent~a~ canservafion programs a? P&j% Power and Light Company: Models, forecasts and gssessments. Paper presented at EPRI Workshop on Measuring Effects of Utility Conservation Pm-

grams. Columbus, OH, February 1982.

5.

7.

8,

9.

10.

Il.

BURNETT, T. Weaiker ud~~strne~t rnetkQd~~~~ used by PGE in We~rherizQtio~ assessment analysis. Portland General Elec-

tric, March 1981. (b)

BURNETT, T. Weatheribon within single-family residences: Report i. Portland General Electric, July 1981. (c)

BURNETT, T. 34ewwing ~e~~~~~~~~i~u~ effe~fj~e~e~: Portland General Electric Company’s experience. Paper

presented at EPRI Workshop on Measuring Effects of Utility Conservation Programs. Columbus, OH, February 1982.

CROFT, J. f. Resident&i weatkerizarion evcrluatian: Actual vs, estimate. Puget Sound Power and Light, January 1982.

BRADLEY, R., & SHAFFER, J. C. Eva/u&ion of SeattIe Ciiy Light’s residential insuintinn program. Seattlr City Light,

February 19X 1.

OLSEN, M., & CLUETT, C., Evafuuation of the Seartfe City tight’s nejgk~orkood energy conservation program. Battefle Human Affairs Research Center, October 1979.


12. WEISS, C. & NEWCOMB, T. Evaluation of the home energy check program. Seattle City Light, October 1981. (draft)

13. GMA RESEARCH CORPORATION. Weatherization barriers survey summary. Puget Sound Power and Light, December 1981. (draft)

REFERENCES

BERRY, L., SODERSTROM, J., HIRST, E., NEWMAN, B., & WEAVER, R. Review of Evaluations of Utility Home Energy Audit Programs (ORNL/CON-58). Oak Ridge, TN: Oak Ridge National Laboratory, March 1981.

BERRY, L., Residential conservation program impacts: Methods of reducing self-selection bias in the interpretation of residential conservation program impacts. Evaluation Review, December 1983.

(ORNL/CON-94, Vol. I). Oak Ridge, TN: Oak Ridge National Laboratory, April 1982.

HIRST, E., BRONFMAN, B., GOELTZ, R., TRIMBLE, J., SC LERMAN, D. Evaluation of the BPA Res~de~t~a~ Weatherization Pilot Program (ORNLICON-124). Oak Ridge, TN: Oak Ridge Na- tional Laboratory, 1983. (Draft)

ROGERS, E. M., & SHOEMAKER, F. F., Communication of in- novations. New York: The Free Press. 1979.

HIRST, E., BERRY, L., 3RONFMAN, B., JOHNSON, K., TEPEL, R., & TRIMBLE, J. Evaluation Plan for the Bonneville Power Adminktration Residential Energy Conservation Programs

SODERSTRO,~, J., BERRY, L., & HIRST, E., The use of metaevaluation to plan evaluations of conservation programs. Evalua- tion and Program Planning, 1981, 4, 113-122.

Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Documents

Transcript of Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review