NJ Comfort Partners Evaluation

Jackie Berger

August 21, 2014

Program Goals And Approach

Goals•Cost-effective energy savings•Improved bill payment•Target assistance program participants•Improve comfort, health, and safety

Approach

•Joint delivery for electric and gas customers•Comprehensive measure installation•Comprehensive customer education

2

NJ Comfort PartnersUtility Partners

3

Electric Utilities

Atlantic City Electric

Jersey Central Power & Light

PSE&G

Evaluation Questions

1. To what extent does the program achieve its goals?

2. What opportunities are available for increased effectiveness or reduced costs?

3. How can the program be refined to achieve improved outcomes?

5

Evaluation Overview

Tracking Process Affordability

Impact

Energy Saving Protocols

6

Tracking System

Information Needs

• Management• Reporting• Operations• Evaluation

Elements to Review

• Program reports• Protocol savings

calculations• Data fields• System use

Assessment Issues

• Content• Quality• Sufficiency• Accuracy• Efficiency

• Invoices• Messaging• Retrieving job data• Job file attachments• BPU reporting• Management reporting

Functions

• Messaging capabilities• Ad hoc reports require request• No ability to import or export data• Balancing accuracy checks and

complexity• Data timeliness• Electronic audit tool• Improved attachment feature

Challenges and Needs

7

Process Research

Background and Procedures Review

Needs Assessment

Manual

Marketing

Staff Training

Energy Education

Data Forms

Utility Interviews

6 Utilities

Managers and Staff

Program Design

Challenges and Successes

Contractor Interviews and On-

Site Assessment

5 Prime Contractors

Quality Control

Contractor

Program Procedures

Data Management

On-Site

Observations Audits

InstallationsFinal

Inspections

Inspections of Completed Jobs

Needs Assessment

8

Service TypeLow-Income Households with Bills

Low-Income Households

with High Bills

Percent of Low-Income Households with High Bills

AllMeet all Eligibility

Criteria

Electric Baseload 623,382 313,335 50% 41%

Electric Heating 92,871 22,417 24% 19%

Gas Heating 462,013 255,523 55% 53%

Low-Income Households

Number Percent

Income at or below 225% 812,370 100%

Heat with Gas or Electric 693,604 85%

Direct Payment of Electric or Gas Bill 730,725 90%

Eligible Housing Type 631,109 78%

Meet All Eligibility Criteria 503,242 62%

Observe

• Systematic and consistent approach

• Use of detailed data collection forms

• Quality rating scales

• Experts trained to conduct assessment

• Quantitatively document findings

ObservedAudits-80 Installs-41

Final Inspections-13

9

Observation Procedures

• Purpose: Observe and record process• Observers should not train, recommend, correct, or otherwise influence the process• Observers should only intervene if they see serious issues

Inspect

• Diagnostics – Tests include blower door, ducts, safety testing, IR, and appliance metering

• Measures – Inspect all installed measures for quality and completeness

• Missed Opportunities – Identify any measures that could have been installed, but were not.

• Client Interview – Discussion of health, comfort, safety issues, and client-related factors

InspectedRepresentative Sample

of 288 Jobs

Inspection Procedures

• Selected from customer survey respondents• Work completed approximately one year ago• $50 incentive offered to participants

10

Observation Assessments

11

Action Completeness of Inspection

1 Auditor inspected every accessible room

2 Auditor measured and recorded dimensions of building

3 Auditor did outside home walk-around

4 Auditor physically checked insulation levels in the attic(s), walls, and basement/crawl

5 Auditor talked with customer about comfort issues

6 Auditor talked with customer about energy issues

7 Auditor talked with customer about health and safety issues

8 Auditor assessed all accessible attics

9 Auditor assessed all accessible basement/crawl space areas

10 Auditor inspected the garage

11 Auditor conducted blower door test

Observation Assessments

12

Action Quality of Air Sealing Full

Scale

Exclude Safety Issues

Impact Usage Scale

1 Fire rated materials used around chimneys, flue pipes, recessed lighting, etc. ü ü  

2 Materials integrated with existing finishes ü ü  

3 Used supply-air respirators when using two-part foams ü    

4 Blower door used to target air sealing ü ü ü

5 Blower door and smoke used to verify air sealing effectiveness ü ü ü

6 Caulk not used to seal gaps larger than 1/8” ü ü ü

7 Large holes and gaps covered with rigid materials ü ü ü

8 Area cleaned before caulk or foam applied ü ü ü

9 All material scraps and packaging removed from home ü ü  

10 Materials installed consistently and are uniform in appearance ü ü  

11 Gloves and eye protection used when cutting, or using caulk or foam ü    

12 Used hard hats when working in confined spaces ü    

13

Affordability Research

Customer Survey

• Understanding• Energy Education• Impact• Satisfaction

Impact

• Bills• USF Subsidy• Energy Burden• Payment

Findings

• Program serves needy households including veterans, customers receiving assistance, and the unemployed

• Some contractors provide more effective education than others

• Customers are most interested in reducing bills

• Small but positive impacts on energy affordability

• Positive impact on ratepayers, as USF subsidy declined

Customer Feedback

14

All

Implementation Contractor

1 2 3 4 5

Surveys 977 459 203 194 101 20

Provider Reviewed Energy Bill 37% 43% 35% 24% 34% 45%

Provide Explained how Energy Use is Measured 48% 50% 50% 40% 47% 45%

Customer Reported Good Understanding Of Energy Bill 78% 81% 79% 74% 70% 80%

Contractor Provided Written Action Plan 54% 58% 58% 44% 46% 45%

Contractor Provided Estimate of Money Savings from Actions 27% 27% 32% 18% 27% 30%

Customer Took Energy Saving Actions 60% 64% 60% 56% 52% 70%

15

Impact Research

Program Data

Analysis

• Customer Characteristics• Housing• Measure Installation• Measure Costs• Health and Safety Costs• % in Spending Tiers• Inspection Pass Rates• Inspection Failure Reasons

Usage Impact

• Data from 6 Utilities• Weather Normalized• Comparison Group

Adjusted• All Treated Customers

with Sufficient Data• Savings Realization Rates• Cost Effectiveness

Protocol Savings Category

Treatment Group

# %

CFL 5,100 79%

Air Sealing 4,201 65%

Hot Water 3,926 61%

HVAC 2,991 46%

Refrigerator 2,797 43%

Thermostat 2,436 38%

Duct Sealing 2,061 32%

Insulation 2,029 32%

  Pre Post Change Measured

Treatment Group

Year Before

Services

Year After Services

After - Before

Program Impact

and Other Factors

Comparison Group

2 Years Before

1 Year Before

2 Years Before – 1 Year Before

Other Factors

Treatment - Comparison 2009-2011 2010-2012   Program

Impact

16

Energy Saving Protocols

Review and Verify

Recommen

ded Chang

esProtocols for Additional

Measures

Engineering Estimates