Evaluation Guidelines: Put to the Test in NY...Evaluation Guidelines: Put to the Test in NY William...
13
Evaluation Guidelines: Put to the Test in NY William P. Saxonis New York State Department of Public Service ACEEE/CEE 2013 Market Transformation Symposium Washington DC March 26,2013 1
Transcript of Evaluation Guidelines: Put to the Test in NY...Evaluation Guidelines: Put to the Test in NY William...
Evaluation Guidelines:
Put to the Test in NY
William P. Saxonis
New York State Department of Public Service
ACEEE/CEE 2013 Market Transformation Symposium
Washington DC March 26,2013
1
Energy Efficiency Portfolio Standard (EEPS)
The PSC established EEPS in June 2008 with a goal of reducing electricity usage by 15% from the 2015 forecast
and a comparable efficiency goal for natural gas
Key Actions:
Approved about 100 new energy efficiency programs, increased the role of utilities in program administration and more than tripled annual program expenditures
Made rigorous evaluation a PSC priority
Increased evaluation funding by 150% and expanded the oversight role of DPS Staff
2
Evaluation Guidelines - A Key First Step
The Commission required Staff to issue Evaluation Guidelines within 45 days of the order establishing the EEPS
The prime objective was to build a solid foundation to meet the evaluation challenges of the expanding size and scope of the New York’s energy efficiency effort
The Evaluation Guidelines address:
-- Evaluation methodology (e.g., statistical rigor)
-- Ethical and operational standards
-- Data collection
-- Cooperation among interested parties
.
3
Lessons Learned
Guidelines are guidelines, not cookbooks!!
A “back to basics approach” works best
Emphasize avoiding deficiencies in basic
evaluation planning and design, such as a
failing to collect the “right data”
4
Lessons Learned
Balance tough standards with the flexibility to
permit evaluators to use the most reliable,
practical and cost-effective methodology
Address key issues - e.g., the 90/10 conundrum
Link evaluation rigor with evaluation priority
Avoid a one size fits all approach!
5
Characteristics of Successful Guidelines
Understandable
Relevant
Embraced
Used
Evolving
Associated with a strong support mechanism …Things don’t just happen
6
The Evaluation Advisory Group (EAG)
Created to advise the Commission and DPS
staff on key evaluation issues
A diverse group of energy professionals
Plays a critical role in encouraging
communication and cooperation by creating a
forum where members can discuss concerns,
share ideas, and solve problems
Communication is key!
7
Listen!!!
Duke Ellington considered himself
"the world's greatest listener"
8
Duke Ellington (1899-1974)
American Genius
Reviewing Key Evaluation Documents Staff reviews evaluation plans and key
deliverables such as survey instruments, sampling designs and draft reports
A time consuming process, but worth the effort
Check for Guideline compliance
"A stitch in time may save nine“
9
Spillover -- A Tool To Measure
Market Transformation
Recent Guidelines Address Spillover:
-- Preliminary methodological questions
-- Level of evaluation rigor
-- Avoiding double counting
-- Calculation of spillover at the program and market Levels
-- Sampling, uncertainty, statistical precision
-- Deemed approaches
10
Conclusions- It Isn’t All About
Equations
Even if we obtain methodological perfection, we need to
better communicate the results
Program managers need to understand that if
evaluation results are below expectation, it is not a “bad
evaluation” provided there is guidance for program
improvement
Think more creatively!
11
Organized Creativity
Duke Ellington had an “insatiable need to break
old rules, write new ones, break new ground,
and to be creative and innovate.”
Dave Brubeck noted how Duke
thought “out of the box….”
12
More Information
DPS Web Site:
http://www.dps.ny.gov/
Visit the DPS EEPS Evaluation Web Section!
My E-mail address:
Thank You!
13
