Experimental Design and Logic Models -...
Transcript of Experimental Design and Logic Models -...
![Page 1: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/1.jpg)
pg. 5
Experimental Design
and Logic Models
![Page 2: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/2.jpg)
pg. 6
Three major components of any behavior pilot
Target Population
Target Behavior to Change
Target Intervention
![Page 3: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/3.jpg)
pg. 7
Starting with a target population
Target Population
Target Behavior to Change
Target Intervention
University
students in
sub-metered
dorms
![Page 4: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/4.jpg)
pg. 8
Starting with a target intervention
Target Population
Target Behavior to Change
Target Intervention
Endorsement
from trusted
messenger
![Page 5: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/5.jpg)
pg. 9
Starting with a target behavior to change
Target Population
Target Behavior to Change
Target Intervention
Select a green energy provider
![Page 6: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/6.jpg)
pg. 10
Three major components of any behavior pilot
Target Population
Target Behavior to Change
Target Intervention
Are there any
characteristics of this
population that limit the
types of behaviors you
can ask to be changed?
Does your target
population limit how we
can measure energy
reduction due to our
interventions?
Can I isolate and measure the specific
target behavior I want to change?
What interventions are best suited to
change my target behavior?
Will my target
intervention work well
with the population I
must use?
Has prior research
shown that my target
intervention is effective
in changing my target
behavior?
![Page 7: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/7.jpg)
pg. 11
Three major components of any behavior pilot
Target Population
Target Behavior to Change
Target Intervention
Start
anywhere...but go full
circle!
![Page 8: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/8.jpg)
pg. 12
The two main types of experiments
![Page 9: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/9.jpg)
pg. 13
True Experiments
Randomly split population
into two or more groups
Intervention
Control
Did not reduce
energy
Reduced
energy
![Page 10: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/10.jpg)
pg. 14
True experiments are all about randomization
• Once you have randomly assigned participants to your groups – you can assume they are EQUAL in all ways
• Then...
• You intervene in one or more groups
• but you leave one alone
• Once the experiment is underway...
• You measure the differences between the groups
• Since you started with equivalent groups you can infer CAUSALITY
![Page 11: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/11.jpg)
pg. 15
Aside on Randomization
![Page 12: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/12.jpg)
pg. 16
Control Group (Even Numbers)
![Page 13: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/13.jpg)
pg. 17
![Page 14: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/14.jpg)
pg. 18
Experimental Group (Odd Numbers)
![Page 15: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/15.jpg)
pg. 19
![Page 16: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/16.jpg)
pg. 20
![Page 17: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/17.jpg)
pg. 21
![Page 18: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/18.jpg)
pg. 22
Discussion
![Page 19: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/19.jpg)
pg. 23
When you can’t randomize
![Page 20: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/20.jpg)
pg. 24
Quasi-Experiments
Match groups to be as
equivalent as possible
Intervention
Control
Did not reduce
energy
Reduced
energy
Two different populations
(area, groups, etc.)
Cannot completely
prove causality
![Page 21: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/21.jpg)
pg. 25
Quasi-Experiments are all about making groups as similar as possible
• You want to show that your intervention CAUSED the
change in behavior
• But...
• You cannot infer causality with quasi-experiments
• There will always be doubt!
That’s ok!• You will have to work hard to show that your groups are
as similar as they can be
• Then you can suggest causality
![Page 22: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/22.jpg)
pg. 26
Logic Models
![Page 23: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/23.jpg)
pg. 27
Role of Logic Models
27
A logic model is your
program road map
Where are you going?
How will you get there?
What will tell you that you’ve arrived?
![Page 24: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/24.jpg)
pg. 28
Logic Models Help:
• Develop indicators of program progress• Main effects may be medium or long term
• Communicate program process• Everyone on the same page
• Prioritize EM&V• Do we need to measure everything?
![Page 25: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/25.jpg)
pg. 29
A graphic representation that shows logical
relationships between inputs, outputs, and
outcomes relative to a situation that shows the
theory and assumptions underlying the
program
APHIC HERE.
Step 2
As
su
mp
tio
ns
Inputs Outputs Outcomes
Build Your Logic Model
12/17/2014 29
Tip: Start with the end in mind!
![Page 26: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/26.jpg)
pg. 30
Developing a Logic Model – Assumptions
12/17/2014 30
Summarize program assumptions: what the program will do, how it will do it.
APHIC HERE.
Example for an ENERGY STAR Manufactured Homes program::
If manufacturers are introduced to the program and learn about incentives to reduce costs to manufacture homes above HUD standards and to ENERGY STAR specifications, then manufacturers will modify their standards and methods to build ENERGY STAR Manufactured Homes.
Retailers and other market actors will be educated about the benefits of the ENERGY STAR Manufactured Homes and energy efficient construction.
Market actors will market the homes, in turn educating the public and increasing awareness for the product, transforming the marketplace for New Energy Star Manufactured Homes. Purchasers of new homes will achieve energy and demand savings.
![Page 27: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/27.jpg)
pg. 31
Using logic models
Outcomes
Energy savingsSustained energy
savingsSpillover to other EE
behaviors
Outputs
Number of participantsNumber of actions
takenClick-through rates
Activities
Marketing activities Outreach Behavior Interventions
![Page 28: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/28.jpg)
pg. 32
Using logic models
Neighborhood
Engagement
Energy Savings
Increase in EE behaviors
Activities
Outputs
Outcomes
Free Audits
# Qualified households
identified
# Energy upgrades
identified
![Page 29: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/29.jpg)
pg. 33
Refrigerator/Freezer Recycling Program
The primary objective of the Refrigerator/Freezer Recycling Program is to offer customers an attractive option to encourage early disposal of their functional, inefficient primary or secondary “residential-type” refrigerators and stand-alone freezers.
• The utility has offered a refrigerator/freezer recycling program, using the same contractor and approach, since late 2004. This year the deemed energy savings are being adjusted downward significantly. However, more favorable pricing terms with the contractor is helping to offset this impact so that the overall program cost-effectiveness remains very favorable.
Program Goals:
• The goal of the Program in 2011 is to recycle approximately 4,500 inefficient refrigerators and freezers and achieve 0.327 aMW of electrical savings.
![Page 30: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/30.jpg)
pg. 34
Exercise
• Develop a Logic Model for Refrigerator/Freezer Recycling Program
Exercise:
• What behavior approach could increase
participation?
• Sketch out the logical relationships in a logic
model.
• Think about how you would design an
experiment to test the impact.
![Page 31: Experimental Design and Logic Models - Empowerempowerefficiency.com/wp-content/uploads/2014/12/5... · •Develop a Logic Model for Refrigerator/Freezer Recycling Program Exercise:](https://reader033.fdocuments.in/reader033/viewer/2022042123/5e9e474499708d1a256cebfa/html5/thumbnails/31.jpg)
pg. 35
Using logic models
Activities
Outputs
Outcomes