Comprehensive Usage Reduction Program Evaluation
Affordable Comfort 2007
Jacqueline Berger
Evaluation Objectives
1. Determine the cost-effectiveness of WRAP
2. Develop standard questions so PPL can measure the same criteria in future evaluations
3. Comply with the PUC Order
2
Evaluation Questions
1. What are the goals and how are they met?
2. What are the admin costs? How can they be lowered?
3. How effective is program recruitment?
4. Is the audit mechanism effective?
5. Is the list of program measures comprehensive?
3
Evaluation Questions
6. Is the education process cost-efficient and effective?
7. What is the level of post-inspection and is it appropriate?
8. Does PPL coordinate WRAP with other weatherization programs?
9. What are the energy savings?
4
Evaluation Questions
10. What is the cost-effectiveness of the various agencies?
11. Does PPL provide adequate support and training for contractors?
5
Evaluation Design
1. Background Research
2. Review of Specifications and Procedures
3. Contractor Survey
4. Baseload Observations
5. Full Cost Observations and Inspections
6
Evaluation Design
6. Customer Survey
7. Usage Impacts
8. Payment Impacts
7
Background Research• Goal – develop a complete understanding of
the WRAP procedures and implementation.• Activities
– Interview WRAP managers and staff– Review program documentation– Review program statistics
• Outputs – Understanding for research foundation– Documentation
8
Background Research
• Key Findings– PPL has five managers who oversee WRAP
and other low-income programs in their geographical area.
• Advantage: ability to focus on needs in a particular geographic area
• Disadvantage: many responsibilities – difficult to oversee work of contractors
– PPL spends much effort to come within 4% of expenditure goal (based on PUC requirement).
9
Background Research
• Key Findings– Affordability customers are prioritized.– Otherwise, jobs sent to contractors on a first
come first served basis. – Program coordination barriers
• Long waiting lists for WAP
• Long waiting lists & requirements for gas programs
• Customers who use gas and electric may not have high enough usage to qualify for either program
10
Background Research
• Recommendations– Prioritize customers based on usage.– Track program coordination and provide
incentives for contractors to coordinate services with other programs.
– Continue to introduce technological improvements, such as the web-based measure reporting form.
11
Review of Specifications and Procedures
• Goal – Determine potential effectiveness of measure selection, measure installation, and energy education
• Activities – Assess procedures and forms:– Education specifications– Education forms– Written technical procedures and manuals– Measure installation rates
12
Review of Specifications and Procedures
• Outputs– Recommendations for modifications to:
• Education procedures
• Education forms
• Measure selection guidelines
• Procedures manual
13
Review of Specifications and Procedures
• Education Findings– All participants receive at least one on-site
education visit– Follow-up education is provided at the time of
the inspection or by phone– Remedial education provided to customers
whose usage increases by at least 10% six months after service delivery
14
Review of Specifications and Procedures• Education Recommendations
– Education should be provided at the time of the audit. The homeowner should be present.
– Customer profile should collect information on potentially large opportunities for saving – use of dehumidifiers, use of second refrigerators, lights/appliances left on at all times.
– Action form should prioritize actions by the potential for energy saving in the individual home. Should list top 3-5 actions with estimated $ savings.
15
Review of Specifications and Procedures• Technical Findings and Recommendations*
– Review cost-effectiveness calculations in audit decision trees to reflect current estimates of costs and savings.
• Refrigerators• CFLs
– Water heater wraps and pipe insulation may be more cost effective than water heater replacement.
16*Blasnik & Associates.
Review of Specifications and Procedures
• Technical Findings and Recommendations– Duct sealing in basements should be focused on
safety and comfort.– Blower door guided air sealing: investigate why
only done in 60% of full cost jobs.– WRAP standards and field guide: more concise
program field guide with separate specifications for specific areas may be useful.
17
Contractor Survey• Goal - Assess contractor compliance with program
procedures, and assess problems in program administration.
• Activities– Develop contractor survey instrument
– Determine survey sample
– Send survey to contractors
– Review completed surveys and contact respondents for additional information/clarifications
18
Contractor Survey• Outputs - Understanding of:
• Contractor background and experience
• Support and training provided to contractors
• Usefulness of program forms
• Program implementation procedures
• Joint service delivery with WAP
• Health and safety problems found in homes
• Contractors’ quality control
• Inspection issues
19
Contractor Survey
• Outputs - Recommendations related to:
– Program procedures
– Contractor training and support
– Inspection procedures
20
Contractor Survey
21
# of Contractors
Complete 16
No Response 2
Contractor Survey
22
Staff Training # of Contractors
Observing other service delivery staff 12
Being observed while delivering services
10
Classroom training 8
Affordable Comfort 6
PA WX Classes 3
WX Training Center Classes 2
Testing staff 1
Contractor Survey
23
Staff Assessment # of Contractors
Field observation of WRAP jobs 11
Practical exam 7
Professional certification 7
Written exam 5
Inspection of WRAP jobs 5
Pass Wx Training Center class 1
Web training 1
Contractor Survey
24
PPL Training Ratings
# of Contractors Who Provided Each Rating Mean
Rating1-2 3 4-5
Quality 1 4 10 3.7
Focus 1 4 9 3.6
Level 1 4 9 3.8
Amount 2 7 5 3.2
Overall 1 3 10 3.6
Contractor Survey
25
Joint Delivery of WRAP Services
# Who do Joint Delivery
% of Jobs
Min Max Mean
PA WX 8 0% 100% 16%
Gas Utility 4 0% 100% 7%
County WX 2 0% 10% 1%
Contractor Survey
26
Provide Evening and Weekend WRAP Services
Evenings Weekends
Yes 8 7
No 7 8
No Answer 1 1
Contractor Survey
27
Baseload Audit Procedures
Always Sometimes Never
Describe WRAP 10 1 0
Discuss bill with customer 11 0 0
Discuss H&S with customer 10 1 0
Conduct walkthrough with customer
8 2 1
Provide measure saving estimate 6 5 0
Provide actions savings estimate 5 6 0
Contractor Survey
28
Quality Control Methods
# of Contractors % Of Jobs
Review data collection forms
11 75%
Contact customers by telephone
10 40%
On-Site Inspection 12 41%
Observation 12 32%
Contractor Survey
29
PPL Inspectors
% of Inspections
Resolution
Inspector Fixed
Action Sheet
None Needed
Invoicing mistake 7% 17% 47% 36%
Insulation 6% 0% 83% 17%
Dryer Venting 15% 39% 61% 0%
Education 11% 100% 0% 0%
Contractor Survey
30
PPL Program Ratings
# of Contractors Who Provided Each Rating Mean
Rating1-2 3 4-5
WRAP Specs 0 3 12 4.1
PPL Communication
0 3 12 4.3
Invoicing 0 4 11 4.1
Overall 0 3 12 3.9
Contractor Survey• Recommendations
– Revisit audit forms and determine whether they can be consolidated.
– Require home walkthrough on all jobs.– Formalize a process to respond to action sheets.
31
Baseload Observations• Goals - Understand how well contractors
address opportunities for baseload usage reduction and whether education is effectively provided.
• Activities– Sample design and selection– Observation protocols– Conduct observations– Review findings and synthesis
32
Baseload Observations• Outputs - Recommendations for:
– Additional contractor training– Additional quality control
33
Baseload Observations
34
Visit Introduction
# Of Observations CommentsYes No
Visit expected 10 0
WRAP explained 6 4
Usage reviewed 4 6 2 did later in the visit
H&S discussed 4 6 2 did later in the visit
Comfort discussed 3 7 2 did later in the visit
Baseload Observations
35
Home Walkthrough
#Comments
Yes No
Inspected every room 6 41 auditor did not
do the walkthrough
Systematic inspection 6 4
Discussed electric uses 8 2
Estimate costs of uses 4 6 4 did later in visit
Reinforced costs later 8 2
Baseload Observations
36
Home Walkthrough
#Comments
Yes No
Discussed actions 7 3 1 later in visit
Estimated savings 5 5 3 later in visit
Discussed willingness to take actions
7 3 1 later in visit
Obtained customer commitment
6 4 2 later in visit
Baseload Observations
37
Home Walkthrough
MinutesComments
Min Max Avg.
Length of walkthrough 13 79 41 2 without not included in average
Part on education 13 45 26
Baseload Observations
38
Refrigerator Replacement
#Comments
Yes No NA
Monitored refrigerator 6 2 2 2 new, 1 couldn’t be moved, 1 broken
Replacement 4 5 1
Explored 2-for-1 1 2 7
Baseload Observations
39
Compact Fluorescent Light Bulbs
#
Yes No NA
Discussed all inside lights 4 6
Discussed all outside lights 7 3
Installed CFLs 8 0 2
Asked if satisfied with lighting 8 0 2
Left extra bulbs for customer 0 10
Baseload Observations
40
Energy Education
#
Yes No NA
Energy education visit 10
Engaged customer 9 1
Reviewed measures 7 2 1
Analyzed electric bill 8 2
Discussed appliances 8 2
Baseload Observations
41
Visit Summary
Minutes
Min Max Mean
Length of visit 70 180 119
Rating
Excellent Good FairNot
Acceptable
Overall rating 4 2 2 2
Baseload Observations
• Recommendations– Review WRAP requirements and expectations
with contractors.– Provide education to contractors on the
importance of 2-for-1 swaps in refrigerator replacement.
– Conduct observation of baseload service delivery.
42
Full Cost Observationsand Inspections• Goals - To understand:
– How well contractors address opportunities for electric usage reduction
– Whether correct measures were selected– Extent to which energy education is provided.
• Activities– Sample design and selection– Observation protocols– Review WRAP technical protocols– Conduction observations– Review findings and synthesis
43
Full Cost Observationsand Inspections
• Outputs – Recommendations for:– Procedures– Training
44
Full Cost Observationsand Inspections
45
Visit Introduction
# Of Observations CommentsYes No
Visit expected 6 0
One of the customers was not present for most of the visit.
WRAP explained 4 2
Usage reviewed 4 2
H&S discussed 6 0
Comfort discussed 6 0
Full Cost Observationsand Inspections
46
Diagnostic Testing
# Of ObservationsComments
Yes No NA
Blower door testing 5 1 One of the observations was a mobile home.
Pressure diagnostics 1 4 1
Full Cost Observationsand Inspections
47
Missed Opportunities
1 House still connected to garage and attic.
Solar hot water.
2Remove dropped sealing and install sheetrock for an air barrier.
3 None.
4Insulate entire attic. Would have been difficult, but possible.
Full Cost Observationsand Inspections
48
OverviewVery Good
Good Fair Comments
Data collection accuracy 1 1 2
Unable to duplicate results in one. Not all forms used in another.
Measure selection 1 3
Attic sealed shut in one so work could not be inspected.
Appropriateness of measures 2 2
One hardship case and contractor told to do everything to reduce usage.
Full Cost Observationsand Inspections
49
OverviewExceptional Good Satisfactory Comments
Effort 3 1 Hard to assess one home because customer had moved and home unoccupied. Evaluator’s tests did not match up in
another case.
Quality 1 2 1
Appropriateness 3 1
Overall rating 1 3
Full Cost Observationsand Inspections
• Comments– They were on the right track. They were not
afraid of working hard.– What he did was correct, but he should have
done more air sealing.– He looked at all the right things and asked the
homeowner to clarify things that were not obvious.
50
Full Cost Observationsand Inspections
• Comments– He inspected the entire home and included the
homeowner in the audit.– He did not fill out all the forms and did not do
testing.– He was very friendly and carefully explained
everything to the customer.– He did everything according to the protocol.
51
Full Cost Observationsand Inspections
• Recommendations– Develop one set of forms that is required for all
jobs.– Provide instructions for each form on the back
of the form.– All applicable diagnostic tests should be
required at the audit visit.
52
Customer Survey
• Goals - Understand the participants’– Demographics– Reasons for participation– Understanding of the program– Actions taken to save electricity– Bill payment difficulties– Perceived program impact on electric bills– Perceived program impact on comfort– Satisfaction with the program
53
Customer Survey
• Activities – Develop the survey instrument– Test the survey instrument– Develop a sample plan– Select the survey sample– Conduct surveys– Analyze the data
54
Customer Survey
• Outputs - Recommendations for – Program procedures– Customer education– Contractor training
55
Customer Survey
56
Demographics
Does anyone in your home have a medical condition that requires the use of additional electricity?
Yes 22%
In the past 12 months, was any member of your household unemployed and looking for work?
Yes 34%
Customer Survey
57
Why did you want to receive WRAP?
Reduce electric bills 64%
Improve comfort of the home 20%
Reduce electric usage 9%
Difficult financial situation 6%
Told to enroll 3%
Received new appliances 2%
Customer Survey
58
What are the benefits of WRAP? (Unprompted)
Energy education 36%
Lower electric bills 35%
Lower electric usage 18%
Safer or more comfortable home 11%
New appliances 9%
Improvements to home 6%
Customer Survey
59
What are the benefits of WRAP? (Prompted)
Lower electric bills 88%
Lower electric usage 91%
Energy education 95%
New appliances 86%
Safer or more comfortable home 92%
Customer Survey
60
What is the most important benefit of WRAP?
Lower electric bills 27%
Energy education 19%
Safer or more comfortable home 11%
Lower electric usage 10%
New appliances 10%
Customer Survey
61
How difficult is it for you to pay your PPL bill?
Very difficult 23%
Somewhat difficult 36%
Not too difficult 25%
Not at all difficult 13%
Customer Survey
62
Were you home for the service provider’s visit?
Home for visit 93%
Home for entire visit 85%
Customer Survey
63
Actions to Save Electricity
Provider recommended actions 83%
Provider gave savings estimates 63%
Provider gave written plan of actions 64%
Provider left information 80%
Customer Survey
64
Reduced Usage Of
Lights 72%
Dishwasher 72%
Electric heat 68%
Electric hot water heater 62%
Air conditioner 56%
Electric dryer 55%
Dehumidifier 39%
Customer Survey
65
Actions Taken (Unprompted)
Use CFLs 37%
Turn off lights not in use 18%
Keep doors/windows closed 11%
Purchase energy efficient appliances 8%
Use cold water for clothes washing 6%
Add insulation, air sealing, other measures 6%
Turn off television 2%
Turn off computer 2%
Wash only full loads 2%
Reduce hot water usage 2%
Customer Survey
• Other Actions – Clean/replace ac filters– Heat fewer rooms– Use less electric heat– Stop using an appliance– Reduce length of showers/ reduce baths– Raise refrigerator temperature– Reduce use of dishwasher
66
Customer Survey
67
Change in Comfort
Winter Summer
Improved 40% 32%
Worsened 1% 1%
No Change 58% 67%
Customer Survey
68
Program Satisfaction
Very satisfied 71%
Somewhat satisfied 22%
Somewhat dissatisfied 4%
Very dissatisfied 2%
Usage Impacts1
• Goal - estimate the actual impact of the program on customer electric usage
• Activities– Obtain program measure data
– Obtain electric usage data
– Obtain weather data
– Weather normalize the data
– Compare change for treatment and comparison groups
69Usage impact analysis done by Blasnik and Associates.
Usage Impacts
• Outputs– Estimate of the impact of the program on
energy usage– Estimate of the impact of particular program
measures– Estimate of the effectiveness of different
providers– Estimate of cost effectiveness of service
delivery
70
Usage Impacts
71
Major Measure Installation Rates
BaseloadLow Cost
Full Cost
Refrigerator replacement 50% 62% 44%
Water heater replacement 1% 59% 10%
Air conditioner replacement 18% 36% 13%
Attic insulation 0% 1% 26%
Other insulation 0% 1% 8%
Air sealing with blower door 0% 1% 15%
HVAC work 0% 0% 11%
Usage Impacts
72
Electric Savings Results
Usage Savings
# Pre Post Gross Net Net %
Baseload 659 9,661 9,394 267 836 8.7%
Low Cost 112 10,869 10,633 236 500 4.6%
Full Cost 1,019 17,912 17,129 783 1,767 9.9%
Usage Impacts
73
Measure Savings Results
Baseload Program
Savings Cost $/kWh
Refrigerator replacement 777 $662 $0.85
Air conditioner replacement 172 $546 $3.17
Usage Impacts
74
Measure Savings Results
Full Cost Program
Savings Cost $/kWh
Refrigerator replacement 532 $606 $1.14
Attic insulation 766 $882 $1.15
Other insulation 887 $999 $1.13
Windows & doors 457 $1,206 $2.64
Air sealing w/Blower door 378 $288 $0.76
Usage Impacts
75
A (75)
B (102)
C (27)
D (108)
E (22)
F (56)
G (95)
H(107)
I (36)
-1000 0 1000 2000 3000 4000Net Savings (mean ± 90% confidence intervals)
Net Savings by ProviderBaseload Program
Usage Impacts
76
Net Savings by ProviderFull Cost Program
A (166)
B (129)
C (281)
D(99)
E (58)
F (59)
G (122)
H (73)
0 1000 2000 3000 4000Net Savings (mean ± 90% confidence intervals)
Usage Impacts
• Recommendations– Reassess air conditioner replacement targeting
strategy and water heater replacement.– Refrigerator replacement, insulation, and
blower door guided air sealing should be pursued and perhaps expanded.
77
Payment Impacts
• Goal - To estimate the impact of the program on customer bills and payments.
• Activities– Obtain customer bill and payment data– Add up bills and payments in the pre and post
period– Compare change for treatment and comparison
groups
78
Payment Impacts
• Outputs – Estimate of the impact of the program on bills and payments.
79
Payment Impacts
80
Usage Savings
Pre Post Gross Net Net %
Total Bill $1,214 $1,194 -$21 -$118 -9.7%
Total Payments $1,124 $1,179 $54 -$58 -5.2%
Bill Coverage Rate 93% 100% 8% 12% 12.9%There were 1,873 customers in the treatment group and 1,228 customers in the comparison group.
Conclusions
• Several types of evaluation activities
• Each research activity brings a different set of information
• Do they tell the same story?
• Synthesis
• Recommendations that can be implemented
81
Top Related