Post on 24-Feb-2016
description
Washington State Low Income Weatherization Program Evaluation Calendar Year 2011DRAFT ResultsPrepared by:Rick Kunkle
July 2013
Purpose of EvaluationIdentify and document Weatherization Program outcomes, benefits and costs to:• Assure prudent use of funds (accountability)• Improve the quality and effectiveness of program
services• Assess progress toward Weatherization Program
outcomes as measured by key performance measures
Weatherization Program OverviewServes low income families by installing energy efficiency measures and making health and safety improvements and necessary repairs• Services provided in Washington since 1977• Managed by the Housing Improvement and
Preservation Unit in the Department of Commerce• Commerce contracts with 28 local agencies to
deliver services • Funded by a mix of state and local sources• The American Recovery and Reinvestment Act
(ARRA) significantly increased funding, primarily in 2010 and 2011
Previous Evaluation Findings• Oak Ridge National Laboratory completed an impact
evaluation for Washington in 2001 showing energy savings compared favorably with other states
• WSU Energy Program completed Washington evaluations for 2006 and FY2010• Benefits exceeded costs• Production increased significantly in FY2010 from
ARRA funding• Recommendations for improving data collection,
estimates of benefits, and cost tracking and allocation
• Commerce has made investments and progress in addressing evaluation recommendations
Evaluation ApproachThe evaluation covers calendar year 2011• Project Data: from the Weatherization information
Data System (WIDS) for projects with final inspections in 2011
• Program Data: from Commerce including expenditures, agency work plans, and historical data
Bottom Line Results for 2011• Expenditures were $48 million. This is more than
twice 2005-2008 annual expenditures. Over half was from ARRA
• Production was 7,451 units. This is a little less than the peak in 2010, but still more than twice pre-ARRA production. Two-thirds of production was multi-family units
• Over 12,000 people lived in housing that was weatherized
• On average, 9 improvement measures were installed in each weatherized unit
• Estimated energy savings is $1.17 million/year. On average this is $157/unit. Other benefits are estimated to be $198/unit
• The Program benefit-cost ratio is estimated to be 1.12, meaning that benefits slightly exceed costs
What is in this Presentation
• Program Delivery• Program Benefits• Program Costs• Program Cost-Effectiveness• Summary and Recommendations
Key Findings – Program Delivery
Historical Production Trends
Monthly Production Trends
Production by Agency in 2011
Agency Service Delivery
Type of Agency
Delivery Method
# of agencies % of FY10 production
Funding Sources
Number of Funding Sources
Project Delivery Time
Average Days to Completion by Agency
Heating Fuel
Weatherization Measures by Unit
Weatherization Measure Categories
Top Weatherization Measures
Top Weatherization Measures by Building Type
Tier 2 Weatherization Measures
Household Demographics
Household Poverty Level
Accomplishments, Challenges, Recommendations
Each year, local agencies produce a work plan that summarizes their processes, procedures, accomplishments, and challenges• Accomplishments:
• Energy savings, comfort, health and safety and housing repair benefits of their weatherization work
• Partnerships, cooperation and referrals• Challenges
• Decreasing funding along with increasing costs from wage and other requirements (reporting, monitoring, training, etc.)
• Recommendations• Clearer communication and more transparent decision-
making about funding availability and allocations• Streamlining program requirements
Quality Assurance Inspections Specifications Correction Factor
Key Findings – Program Benefits
Energy Savings Analysis in WIDSDeemed savings estimates:• Developed savings coefficients by measure, building
type, fuel source, and heating zone• Used savings coefficients with measure data in WIDs in
savings calculations for each measure• Accounted for interactions between measures that save
heating energy• Energy savings are calculated for each weatherization
project with caps and checks• Energy cost savings based on average Washington
energy costs by fuel source• Energy benefits calculations use weighted average
measure life – 34 years
Total Energy SavingsFuel Source Building Type
Total Energy Savings by MeasureCategory Measure
Energy Savings per Unit by Agency
Energy Cost SavingsFuel Source Building Type
Energy Cost Savings per Unit
Other Benefits CalculationsThis is the same approach used for the FY2010 Evaluation• Utility benefits > estimated from best practices review
• Reduce delinquent bills and bad debt write-off• Service shut offs and reconnects
• Participant benefits > estimated from best practices review• Water/wastewater savings• Increased property value• Fewer moves• Health, safety and comfort benefits
• Societal benefits • Reduce greenhouse gas and pollutant emissions >
directly estimated
• Improvements to the economy > estimated from evaluation of Weatherization Program for Pacific Power
Program Benefits
$/Household/Yr Mid Percent of Total
Utility 16 4%
Participant 103 29%
Societal - Economic 60 17%
Societal - Emissions 19 5%
Other Total 198 56%
Energy 157 44%
Total 355 100%
Key Findings – Program Costs
Program Expenditures by Fund Source
Program Expenditures by Category
Direct Project Cost by Fund Source
Direct Project Cost by Measure Category
Direct Average Unit Cost by Agency
Distribution of Direct Unit Cost
Cumulative Direct Unit Cost Curve
High Cost Projects
Average Direct Unit Cost
Total Direct Cost Expenditures
Total Unit Cost Estimate
Why are per unit costs higher in 2011 compared to FY2010?
Many of the reasons could be associated with ARRA• More comprehensive weatherization• Wage requirements• Reporting requirements• Quality assurance requirements• Others?
Key Findings – Program Cost Effectiveness
Per Unit Cost Divided by Energy Savings
Simple Energy Payback
Cumulative Direct Unit Cost and Energy Savings
Cost-Effectiveness Analysis Approach
• Cost-Effectiveness = total benefits per unit/total costs per unit
• Program benefits occur over time and are converted to a present value• 30 year average measure life• 2.3% discount factor (OMB)• 1% fuel escalation factor
• A high, mid and low scenario are considered to account for uncertainty in the benefits estimates
• Alternate approaches with different costs and benefits are also considered
Cost-Effectiveness Analysis Results($/unit costs)
Present Value Mid Low High
Emissions Benefit $421 $369 -
Economic Benefit $1,313 $685 $1,967
Utility Benefit $338 $76 $674
Participant Benefit $2,233 $920 $4,588
Total Non-Energy $4,305 $2,050 $7,229
Energy Benefit $3,934 $2.992 $4,571
Total Benefit $8,239 $5,042 $11,800
Total Cost $7,362 $7,362 $7,362
Benefit-Cost Ratio 1.12 0.68 1.60
Cost-Effectiveness Alternate Approaches
($/unit costs)Present Value Energy Only Plus
No Admin or T&TA $
Direct Cost Only
Plus No H&S and WRR
Emissions Benefit 0 0 0 0
Economic Benefit 0 0 0 0
Utility Benefit 0 0 0 0
Participant Benefit 0 0 0 0
Total Non-Energy 0 0 0 0
Energy Benefit $3,934 $3,934 $3,934 $3,934
Total Benefit $3,934 $3,934 $3,934 $3,934
Total Cost $7,362 $6,414 $4,780 $3,724
Benefit-Cost Ratio 0.53 0.61 0.82 1.06
Summary and Recommendations
Some Interesting Results for 2011• Higher production during ARRA is due to large multi-
family units• Top 9 producers account for 80% of production• City and county government agencies account for over
half the production, but less than half the agencies• Compared to FY2010 there is evidence of more
comprehensive weatherization• Ceiling, wall, and floor insulation and air sealing account
for 55% of the estimated energy savings• Large multi-family units account for 60% of the units,
but only a quarter of the energy savings and a quarter of total direct costs
• The 10% of highest cost units account for 32% of total direct costs and 27% of total energy savings
Conclusions to Think About• The total program unit cost estimate
increased in 2011 compared to FY2010• Large multi-family units contribute to higher
production, more people served, and lower costs, but they also have fewer measures installed and produce less energy savings
• High cost units contribute significantly to total Program costs, but they also have more measures installed and greater energy benefits
• Energy cost benefits are less than other benefits
• While high production agencies had lower unit costs, they did not have lower unit costs/energy savings
Recommendations• We need to develop better ways to track
Program expenditures for Commerce• The Program unit cost estimate includes an
estimate of local Program operations costs that did not go directly to unit weatherization. We need to check the accuracy of this estimate and identify what these cost are for and how to track them
• Identify ways to reduce the trend towards higher costs, which reduces Program cost-effectiveness if benefits do not increase
• How do we measure success? Identify key Program metrics and use WIDS data or other sources to document Program success
• Consider looking at the benefits from high cost units
Next Steps
• Review and discuss the evaluation results• Make needed revisions and corrections to the
evaluation • Complete an executive summary for the
evaluation (text document)• Share the evaluation with the weatherization
network and stakeholders• Apply the experience from the 2011
evaluation to plan for and conduct the 2012 evaluation